{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# NFL Big Data Bowl 2022: Punt Return Yards Over Expected (PRYOE)","metadata":{}},{"cell_type":"markdown","source":"***","metadata":{}},{"cell_type":"markdown","source":"## Outline of Notebook\n\n1. Introduction\n2. Exploratory Data Analysis\n3. Feature Engineering\n4. Baseline Model and Model Tuning with GridSearch CV\n5. Final Model Creation\n6. Production Run\n7. Results\n8. Discussion\n9. Acknowledgements","metadata":{}},{"cell_type":"markdown","source":"<br>\n<br>","metadata":{}},{"cell_type":"markdown","source":"***","metadata":{}},{"cell_type":"markdown","source":"# Introduction\n\n- **Name: Sean Sullivan**\n- **Affiliation: URAM Analytics**\n- **Twitter Handle: @URAM_Analytics**\n- **Website: https://www.uramanalytics.com/**","metadata":{}},{"cell_type":"markdown","source":"With the NFL Big Data Bowl 2022's focus on Special Teams, I wanted to focus my efforts on creating a metric and corresponding analysis to evaluate punt returns. Punts that result in a return only account for about 38% of punts from the 2018, 2019, and 2020 seasons. So while the plays are fairly infrequent, they could have a large impact on a game. \n\nTo do this, I modeled how many yards, on the x axis, that a returner would gain on a frameID level, for a play that was a punt and had a return. To get the data prepared for modeling, I went through a very thorough feature engineering process that is detailed below. With the data in hand, I was able to generate a very accurate model to predict how many yards that a punt returner would be expected to gain, given the current situation as of that frame. By \"current situation\" I mean the location of the returner, how fast they are moving, their direction and orientation, how far away kick return and kick defending players are from the returner and their movement attributes (speed, acceleration, etc), and game specific attributes (time, score, etc). The model was trained on data from 2018 and 2019 and the final model was ran on 2020 data to generate the metric results.\n\nTo create the metric, I subtracted the predicted yards gained in a frame from the actual yards gained in a frame (essentially the error/residual). A positive value would suggest that the returner gained more yards than \"expected\" whereas a negative value would suggest that the returner gained less yards than expected. I believe that this is appropriate given how accurate the model was. If it was less accurate, then I do not think this would have been worth the effort. Since this was done on a frame level, I aggregated the results by Game and Play to know the results for a given punt return play. \n\nWith the results in hand, evaluation could be done on a play level, a player level, and a team level. Please continue to read the notebook to learn more! \n\n\n*Note: Not all of the work was completed in python. In fact, all of the feature engineering was conducted in GCP's BigQuery. Links to all queries are provided in the appropriate sections.*","metadata":{}},{"cell_type":"markdown","source":"<br>\n<br>","metadata":{}},{"cell_type":"markdown","source":"***","metadata":{}},{"cell_type":"markdown","source":"# Load Data and EDA","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport numpy as np\nimport time\n\nfrom sklearn.model_selection import train_test_split\nimport xgboost as xgb\nfrom sklearn.model_selection import GridSearchCV\nfrom sklearn.metrics import r2_score,mean_squared_error, mean_absolute_error\n\npd.set_option('display.max_columns', None)\npd.set_option('display.max_rows', None)","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:30:40.314293Z","iopub.execute_input":"2022-01-02T17:30:40.314822Z","iopub.status.idle":"2022-01-02T17:30:41.806860Z","shell.execute_reply.started":"2022-01-02T17:30:40.314710Z","shell.execute_reply":"2022-01-02T17:30:41.805953Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Load plays data and perform basic exploration of punt data\n\nplays = pd.read_csv('../input/nfl-big-data-bowl-2022/plays.csv')","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:30:43.703237Z","iopub.execute_input":"2022-01-02T17:30:43.703558Z","iopub.status.idle":"2022-01-02T17:30:43.885217Z","shell.execute_reply.started":"2022-01-02T17:30:43.703525Z","shell.execute_reply":"2022-01-02T17:30:43.883826Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# What is the shape of the plays file (rows x columns)\n\nplays.shape","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:30:45.865356Z","iopub.execute_input":"2022-01-02T17:30:45.865675Z","iopub.status.idle":"2022-01-02T17:30:45.875058Z","shell.execute_reply.started":"2022-01-02T17:30:45.865641Z","shell.execute_reply":"2022-01-02T17:30:45.873836Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# What are the columns present?\n\nplays.columns","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:30:47.475019Z","iopub.execute_input":"2022-01-02T17:30:47.476295Z","iopub.status.idle":"2022-01-02T17:30:47.483946Z","shell.execute_reply.started":"2022-01-02T17:30:47.476229Z","shell.execute_reply":"2022-01-02T17:30:47.482819Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<br>","metadata":{}},{"cell_type":"code","source":"\"\"\"\nUsing this guide, we know what each of the columns are, but will provide a brief exploration (https://www.kaggle.com/c/nfl-big-data-bowl-2022/data)\n\nLet's evaluate a few key variables from the plays table\n\"\"\"\n# Evaluate playResult, but only on a punt. We do this by filtering the dataframe ('plays')\n\npunt_plays = plays[plays['specialTeamsPlayType'] == 'Punt']\n\npunt_plays[['playResult']].describe()","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:30:50.642888Z","iopub.execute_input":"2022-01-02T17:30:50.643490Z","iopub.status.idle":"2022-01-02T17:30:50.684837Z","shell.execute_reply.started":"2022-01-02T17:30:50.643457Z","shell.execute_reply":"2022-01-02T17:30:50.683998Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Visualize the distribution of playResult for specialTeamsPlayType = 'Punt'\n\nplt.hist(punt_plays['playResult'], bins=20)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:30:55.808737Z","iopub.execute_input":"2022-01-02T17:30:55.809043Z","iopub.status.idle":"2022-01-02T17:30:56.086121Z","shell.execute_reply.started":"2022-01-02T17:30:55.809013Z","shell.execute_reply":"2022-01-02T17:30:56.084889Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Interpretation:** This shows us that on the 5,991 plays in this table that are a specialTeamsPlayType = 'Punt', the kicking team is gaining about 40 yards on the play. The median value (50th Percentile) is nearly identical to the mean, the data is nearly symmetrical. The histogram confirms this, and shows outliers to the left (slight left skew...the distribution is non-normal). ","metadata":{}},{"cell_type":"markdown","source":"<br>","metadata":{}},{"cell_type":"code","source":"# What about this, but from the point of view of the returning team? For this, we will evaluate 'kickReturnYardage'\n\npunt_plays[['kickReturnYardage']].describe()","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:30:58.463891Z","iopub.execute_input":"2022-01-02T17:30:58.464888Z","iopub.status.idle":"2022-01-02T17:30:58.480034Z","shell.execute_reply.started":"2022-01-02T17:30:58.464832Z","shell.execute_reply":"2022-01-02T17:30:58.479110Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Visualize the distribution of kickReturnYardage for specialTeamsPlayType = 'Punt'\n\nplt.hist(punt_plays['kickReturnYardage'], bins=20)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:31:01.089284Z","iopub.execute_input":"2022-01-02T17:31:01.089611Z","iopub.status.idle":"2022-01-02T17:31:01.339356Z","shell.execute_reply.started":"2022-01-02T17:31:01.089577Z","shell.execute_reply":"2022-01-02T17:31:01.338400Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Interpretation:** This shows us that on specialTeamsPlayType = 'Punt', the returning team is gaining about 9 yards on the play. Similar to playResult, the median value (50th Percentile) is close to the mean. The histogram confirms shows outliers to the right (slight right skew...the distribution is non-normal). This result is not especially surprising given the nature of punt returns. ","metadata":{}},{"cell_type":"markdown","source":"<br>","metadata":{}},{"cell_type":"code","source":"# What kind of results are associated with punt plays? We will evaluate specialTeamsResult with the same filtered dataframe as above. \n# Since it is categorical, we will use a difference method. \n\npunt_plays['specialTeamsResult'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:31:03.485299Z","iopub.execute_input":"2022-01-02T17:31:03.485581Z","iopub.status.idle":"2022-01-02T17:31:03.494930Z","shell.execute_reply.started":"2022-01-02T17:31:03.485550Z","shell.execute_reply":"2022-01-02T17:31:03.494006Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Let's visualize the value count results\n\nax = sns.countplot(x='specialTeamsResult', data=punt_plays, order = punt_plays['specialTeamsResult'].value_counts().index)\nax.set_title('Punt Play Result Count')\nax.set_xlabel('Result Type')\nax.set_ylabel('Count')\nplt.xticks(rotation=90)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:31:05.485971Z","iopub.execute_input":"2022-01-02T17:31:05.486307Z","iopub.status.idle":"2022-01-02T17:31:05.767308Z","shell.execute_reply.started":"2022-01-02T17:31:05.486260Z","shell.execute_reply":"2022-01-02T17:31:05.766312Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Interpretation:** The most common results of a punt play are return, fair catch, and downed. For this project, we are only interested in working with punts that are associated with returns.  ","metadata":{}},{"cell_type":"markdown","source":"<br>","metadata":{}},{"cell_type":"markdown","source":"### What about the tracking data mentioned earlier? What does that look like?","metadata":{}},{"cell_type":"code","source":"# Load 2020 Tracking Data file\n\nfrom google.cloud import storage\n\ntracking_2020 = pd.read_csv('../input/nfl-big-data-bowl-2022/tracking2020.csv')","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:31:08.611372Z","iopub.execute_input":"2022-01-02T17:31:08.611649Z","iopub.status.idle":"2022-01-02T17:31:55.549833Z","shell.execute_reply.started":"2022-01-02T17:31:08.611621Z","shell.execute_reply":"2022-01-02T17:31:55.548820Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tracking_2020.head()","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:31:55.551969Z","iopub.execute_input":"2022-01-02T17:31:55.552442Z","iopub.status.idle":"2022-01-02T17:31:55.579854Z","shell.execute_reply.started":"2022-01-02T17:31:55.552400Z","shell.execute_reply":"2022-01-02T17:31:55.578630Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<br>","metadata":{}},{"cell_type":"markdown","source":"**Let's explore a few key fields: s (speed), dis (distance traveled from prior time point in yards), and event (tagged play details)**","metadata":{}},{"cell_type":"code","source":"# s exploration\n\ntracking_2020['s'].describe()","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:31:55.581668Z","iopub.execute_input":"2022-01-02T17:31:55.582572Z","iopub.status.idle":"2022-01-02T17:31:55.935914Z","shell.execute_reply.started":"2022-01-02T17:31:55.582515Z","shell.execute_reply":"2022-01-02T17:31:55.934835Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Visualize the distribution of s (speed)\n\nplt.hist(tracking_2020['s'], bins=20)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:31:55.938195Z","iopub.execute_input":"2022-01-02T17:31:55.938851Z","iopub.status.idle":"2022-01-02T17:31:56.328707Z","shell.execute_reply.started":"2022-01-02T17:31:55.938806Z","shell.execute_reply":"2022-01-02T17:31:56.327800Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Interpretation:** It's important to understand the scale of this data before interpreting. The tracking data is taken at a frameID level - which is a snapshot of a play in action. It's also important to note that plays do not begin with frameID = 1 on the snap and the last frameID with the end result. On all plays, the snap event is at some later frame and the end event is not the last frame; meaning that there is extra movement recorded before and after a play begins and ends. So, it is to no surprise that most speed values are quite low and that there are a few outliers of very fast player movement. ","metadata":{}},{"cell_type":"markdown","source":"<br>","metadata":{}},{"cell_type":"code","source":"# dis exploration\n\ntracking_2020['dis'].describe()","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:31:56.329956Z","iopub.execute_input":"2022-01-02T17:31:56.330211Z","iopub.status.idle":"2022-01-02T17:31:56.705456Z","shell.execute_reply.started":"2022-01-02T17:31:56.330178Z","shell.execute_reply":"2022-01-02T17:31:56.704561Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Visualize the distribution of dis (distance)\n\nplt.hist(tracking_2020['dis'], bins=20)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:31:57.125315Z","iopub.execute_input":"2022-01-02T17:31:57.125544Z","iopub.status.idle":"2022-01-02T17:31:57.545272Z","shell.execute_reply.started":"2022-01-02T17:31:57.125515Z","shell.execute_reply":"2022-01-02T17:31:57.544286Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Interpretation:** With the added context of the level of granularity that this data provides, it is not a surprise that on a play at the frame level, the yards gained are quite small. Especially when accounting for frames before the ball is snapped. ","metadata":{}},{"cell_type":"markdown","source":"<br>","metadata":{}},{"cell_type":"code","source":"# Let's look at events\n\ntracking_2020['event'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:31:57.546597Z","iopub.execute_input":"2022-01-02T17:31:57.546815Z","iopub.status.idle":"2022-01-02T17:31:59.421401Z","shell.execute_reply.started":"2022-01-02T17:31:57.546789Z","shell.execute_reply":"2022-01-02T17:31:59.420632Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Interpretation:** With being able to see events at the frame level, I used this to only look at punt return plays between the frame where the punt was received and the end event of the play (tackle, touchdown, fumble, etc. ","metadata":{}},{"cell_type":"markdown","source":"<br>\n<br>","metadata":{}},{"cell_type":"markdown","source":"***","metadata":{}},{"cell_type":"markdown","source":"# Feature Engineering ","metadata":{}},{"cell_type":"markdown","source":"I performed all of my data prep and feature engineering in Google Cloud Platform's BigQuery data warehouse. Since none of this was performed in python, I will link you to a text file in my GitHub repository that contains all of the code (with comments) that I used to ultimately create my datasets for model development and production. However, in this section, I will detail my thought process and provide a data dictionary of the final features created. \n\n\nSince this project was focused on modeling returner yards on a punt return, I needed to have the rows contain as much detail as possible to create a robust enough model. With the level of granularity provided by the Tracking files, I knew that this is where I would spend the majority of my efforts. After vigorous exploration, I determined that only focusing on frameID level rows, with the returner's current tracking situation (x, y, s, etc) and each of the other players' data on the same row would be necessary for modeling. To clarify, I needed to know the returner's tracking data and each of the players' tracking data in relation to the returner's at the frameID level. I included some additional game situation variables (like score differential, time situation, etc) to add in context to the play. Below, I provide a high level overview of the process I took and below that, I will provide the file used for training and testing the model.  \n\nLink to Code: https://github.com/seanwsullivan1/NFL-Big-Data-Bowl-2022/blob/main/2022_NFL_BDB_SQL_DataPrep_FeatureEngineering.txt","metadata":{}},{"cell_type":"markdown","source":"<br>","metadata":{}},{"cell_type":"markdown","source":"## SQL Feature Engineering Process:\n\n- **Aggregate all tracking data from 2018, 2019, and 2020 files that were associated with plays that were: SpecialTeamsPlayType = 'Punt' and specialTeamsResult = 'Return'**\n\n\n- **Determine frameID, by gameID and playID where the ball was snapped, where the final event (not a None event), and the frameID one past the final event frameID.** This is necessary because I needed the x value for the next frame after the final event so that I could determine yards gained. This is because the \"situation\" of the frameID at frameID = 2 is the result of what happened in frameId = 1. \n\n\n- **Use the table generated in Step 2 to filter out frameIDs from the table generated in Step 1**\n\n\n- **With the table generated in Step 3, grab the x value from the following frameID and attach it as a new column.** Using the example above, now the x value at frameID = 2 would be on the same row as frameID = 1. \n\n\n- **I wanted to provide movement features for each of the 11 players on the kicking team and the 10 additional players (not including returner) on the returning team.** To do this, I needed to assign an identity at the beginning of the play so they could be followed throughout. I did this by determining if a player was on the kicking or receiving team and labled them \"kicking_NUMBER\" or \"receiving_NUMBER\" based on their sorting on the Y axis at the frameID where the ball was snapped. \n\n\n- **Join all 11 kicking team players' and 10 receiving players' X, Y, S, A, DIS, O, and DIR values to the same row as the returner's for each frameID within a gameID and playID.**\n\n\n- **Create features for: the distance, in yards, of each player in relation to the returner as of that frameID; count, in buckets, the number of kicking team players or returning team players that are within 5, 5-10, 10-15, and 15+ yards from the returner, and the spread of the kicking and returning teams on the x and y axis.** Additional features were created from some of the situational variables such as Quarter, Advantage (which team is winning), Score Differential, and Time Situation. \n\n\n- **With the table generated in Step 7, I: filtered the data to only include frames between punt received and end event, dropped features that would not be needed, created my dependent variable (x_gained_dv), and added team names.** Using the playDirection field and the x value from step 4, I was able to determine if the returner movement was a positive or negative gain on the x axis. \n\n\n- **With the table generated in Step 8: filter out plays that had penalties and split into two tables (one for training/testing the model and one for generating the analysis/metric.** Plays from 2018 and 2019 were used for the model training/testing the 2020 data was saved for production. ","metadata":{}},{"cell_type":"markdown","source":"<br>","metadata":{}},{"cell_type":"code","source":"# Load training data\n\ntraining_data = pd.read_csv('../input/d/seanwsullivan1/nfl-big-data-bowl-2022/NFL_Big_Data_Bowl_Fall_2021_training_data.csv')","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:31:59.422337Z","iopub.execute_input":"2022-01-02T17:31:59.422580Z","iopub.status.idle":"2022-01-02T17:32:00.899059Z","shell.execute_reply.started":"2022-01-02T17:31:59.422552Z","shell.execute_reply":"2022-01-02T17:32:00.897628Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Quick look at the file (transposed)\n\ntraining_data.head().T","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:32:00.902737Z","iopub.execute_input":"2022-01-02T17:32:00.903026Z","iopub.status.idle":"2022-01-02T17:32:00.940341Z","shell.execute_reply.started":"2022-01-02T17:32:00.902997Z","shell.execute_reply":"2022-01-02T17:32:00.939157Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<br>","metadata":{}},{"cell_type":"markdown","source":"## Quick Data Dictionary for Created Features\n\n* **Dis Upated and X Updated**: The dis and x values from the next frameID for a player within a game and play. Dis Updated was not used in final iteration and X Updated was but only in the creation of the dependent variable. NOTE: these are not used as features in modeling. \n\n* **Kicking/Receiving Distance**: Distance, in yards, between player indicated and the returner. \n\n* **Kicking/Receiving S**: Speed, at that frameID, of the player indicated.\n\n* **Kicking/Receiving A**: Acceleration, at that frameID, of the player indicated.\n\n* **Kicking/Receiving O**: Orientation, at that frameID, of the player indicated.\n\n* **Kicking/Receiving Dir**: Direction, at that frameID, of the player indicated.\n\n* **Kicking/Receiving Buckets**: Bucket 1: # of players within 0-5 yards, Bucket 2: # of players within 5-10 yards, Bucket 3: # of players within 10-15 yards, Bucket 4: # of players within 15+ yards\n\n* **X Gained DV**: Number of yards, on x axis, gained (either positive or negative) as a result of the situation of the current frameID. **THIS IS THE DEPENDENT VARIABLE**","metadata":{}},{"cell_type":"markdown","source":"<br>","metadata":{}},{"cell_type":"markdown","source":"**Exploration of Dependent Variable: X Gained DV**","metadata":{}},{"cell_type":"code","source":"# X Gained DV\n\ntraining_data['x_gained_dv'].describe()","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:32:00.942299Z","iopub.execute_input":"2022-01-02T17:32:00.942620Z","iopub.status.idle":"2022-01-02T17:32:00.959677Z","shell.execute_reply.started":"2022-01-02T17:32:00.942566Z","shell.execute_reply":"2022-01-02T17:32:00.958789Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Visualize the distribution of X Gained DV\n\nplt.hist(training_data['x_gained_dv'], bins=20)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:55:44.892784Z","iopub.execute_input":"2022-01-02T17:55:44.893816Z","iopub.status.idle":"2022-01-02T17:55:45.104578Z","shell.execute_reply.started":"2022-01-02T17:55:44.893737Z","shell.execute_reply":"2022-01-02T17:55:45.103219Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from scipy.stats import shapiro\n\n#perform Shapiro-Wilk test\nshapiro(training_data['x_gained_dv'])","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:32:00.961067Z","iopub.execute_input":"2022-01-02T17:32:00.961282Z","iopub.status.idle":"2022-01-02T17:32:00.977221Z","shell.execute_reply.started":"2022-01-02T17:32:00.961256Z","shell.execute_reply":"2022-01-02T17:32:00.976169Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Interpretation:** The mean and median are very close (near 0.2) which suggests data symmetry. Evaluating the histogram, shows a nearly normal appearing distribution. However, using the Shapiro-Wilk test of normality, the test statistic is 0.99 and the p-value is less than 0.05. Thus, we reject the null hypothesis and have sufficient evidence to determine that the dependent variable does not come from a normal distribution. ","metadata":{}},{"cell_type":"markdown","source":"<br>\n<br>","metadata":{}},{"cell_type":"markdown","source":"***","metadata":{}},{"cell_type":"markdown","source":"# Load training/test data, hyper-parameter tuning, and model evaluation\n\nNote: Model was trained/tested on 2018 and 2019 data. The final model was ran on 2020 data and this is what was used to generate the metric results. ","metadata":{}},{"cell_type":"code","source":"training_data = pd.read_csv('../input/d/seanwsullivan1/nfl-big-data-bowl-2022/NFL_Big_Data_Bowl_Fall_2021_training_data.csv')","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:32:00.978894Z","iopub.execute_input":"2022-01-02T17:32:00.979253Z","iopub.status.idle":"2022-01-02T17:32:02.084590Z","shell.execute_reply.started":"2022-01-02T17:32:00.979219Z","shell.execute_reply":"2022-01-02T17:32:02.083630Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Get shape of file\n\ntraining_data.shape","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:48:41.625509Z","iopub.execute_input":"2022-01-02T17:48:41.625837Z","iopub.status.idle":"2022-01-02T17:48:41.631720Z","shell.execute_reply.started":"2022-01-02T17:48:41.625803Z","shell.execute_reply":"2022-01-02T17:48:41.630840Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Split into numeric and categorical variables for evaluation\n\nnum_var = training_data.drop(['gameID', 'playID', 'frameID', 'nflID', 'displayName', 'event', 'dis_updated', 'x_updated',\n                   'punt_received_frame', 'ultimate_id','quarter', 'advtanage', 'team', 'time_situation',\n                   'receiving_team_name', 'kicking_team_name'], 1)\n\ncat_var = training_data[['playDirection', 'quarter', 'advtanage', 'team', 'time_situation']]","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:32:02.086236Z","iopub.execute_input":"2022-01-02T17:32:02.086468Z","iopub.status.idle":"2022-01-02T17:32:02.109667Z","shell.execute_reply.started":"2022-01-02T17:32:02.086440Z","shell.execute_reply":"2022-01-02T17:32:02.108845Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Evaluate Numerical Variables\n\nnum_var.describe().T","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:32:02.111121Z","iopub.execute_input":"2022-01-02T17:32:02.111974Z","iopub.status.idle":"2022-01-02T17:32:02.806808Z","shell.execute_reply.started":"2022-01-02T17:32:02.111925Z","shell.execute_reply":"2022-01-02T17:32:02.805746Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Categorical Variable Exploration\n\nfor i in cat_var:\n    print()\n    print(cat_var[i].value_counts())\n    print('---------------------------------------------')","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:32:02.808283Z","iopub.execute_input":"2022-01-02T17:32:02.808585Z","iopub.status.idle":"2022-01-02T17:32:02.871096Z","shell.execute_reply.started":"2022-01-02T17:32:02.808545Z","shell.execute_reply":"2022-01-02T17:32:02.870022Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Check for Nulls\n\n# Numeric Variable Check\nnum_var.isnull().values.any()","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:32:02.872642Z","iopub.execute_input":"2022-01-02T17:32:02.872980Z","iopub.status.idle":"2022-01-02T17:32:02.898454Z","shell.execute_reply.started":"2022-01-02T17:32:02.872936Z","shell.execute_reply":"2022-01-02T17:32:02.897384Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Categorical Variable Check\ncat_var.isnull().values.any()","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:32:02.901380Z","iopub.execute_input":"2022-01-02T17:32:02.901713Z","iopub.status.idle":"2022-01-02T17:32:02.938103Z","shell.execute_reply.started":"2022-01-02T17:32:02.901679Z","shell.execute_reply":"2022-01-02T17:32:02.937197Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<br>\n<br>","metadata":{}},{"cell_type":"markdown","source":"## Split into X/Y and Train/Test Sets","metadata":{}},{"cell_type":"code","source":"# x variables\n\nx = training_data.drop(['gameID', 'playID', 'frameID', 'nflID', 'displayName', 'event', 'punt_received_frame', \n             'ultimate_id', 'dis_updated', 'x_updated', 'x_gained_dv', 'receiving_team_name', 'kicking_team_name'],1)","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:32:02.940105Z","iopub.execute_input":"2022-01-02T17:32:02.940442Z","iopub.status.idle":"2022-01-02T17:32:02.967270Z","shell.execute_reply.started":"2022-01-02T17:32:02.940397Z","shell.execute_reply":"2022-01-02T17:32:02.966354Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# y variable\n\ny = training_data['x_gained_dv']","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:32:02.968862Z","iopub.execute_input":"2022-01-02T17:32:02.969294Z","iopub.status.idle":"2022-01-02T17:32:02.977682Z","shell.execute_reply.started":"2022-01-02T17:32:02.969246Z","shell.execute_reply":"2022-01-02T17:32:02.976623Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Get dummy variables for categorical x variables\n\nx = pd.get_dummies(x)","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:32:02.979496Z","iopub.execute_input":"2022-01-02T17:32:02.979779Z","iopub.status.idle":"2022-01-02T17:32:03.064093Z","shell.execute_reply.started":"2022-01-02T17:32:02.979729Z","shell.execute_reply":"2022-01-02T17:32:03.062855Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Check x columns\n\nlist(x.columns)","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:32:03.066620Z","iopub.execute_input":"2022-01-02T17:32:03.067344Z","iopub.status.idle":"2022-01-02T17:32:03.074802Z","shell.execute_reply.started":"2022-01-02T17:32:03.067301Z","shell.execute_reply":"2022-01-02T17:32:03.074198Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Print shapes of x and y\n\nprint(x.shape)\nprint(y.shape)","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:32:03.075780Z","iopub.execute_input":"2022-01-02T17:32:03.076309Z","iopub.status.idle":"2022-01-02T17:32:03.091212Z","shell.execute_reply.started":"2022-01-02T17:32:03.076273Z","shell.execute_reply":"2022-01-02T17:32:03.089396Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Train/Test Split\n\nx_train, x_test, y_train, y_test = train_test_split(x,y, test_size=0.3, random_state = 21)\n\nprint(x_train.shape)\nprint(x_test.shape)\nprint(y_train.shape)\nprint(y_test.shape)","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:32:07.020482Z","iopub.execute_input":"2022-01-02T17:32:07.021191Z","iopub.status.idle":"2022-01-02T17:32:07.104197Z","shell.execute_reply.started":"2022-01-02T17:32:07.021139Z","shell.execute_reply":"2022-01-02T17:32:07.103188Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<br>\n<br>","metadata":{}},{"cell_type":"markdown","source":"## Baseline XGBoost Regressor Model","metadata":{}},{"cell_type":"markdown","source":"Before embarking on tuning the model, it's important to understand what the baseline performance is. We evaluate that here before moving on to tuning the model. ","metadata":{}},{"cell_type":"code","source":"# Run model with timer\n\nstart = time.time()\n\nmodel = xgb.XGBRegressor(seed=21)\nmodel.fit(x_train, y_train)\n\nprint('Runtime: ', time.time() - start)","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:32:09.498035Z","iopub.execute_input":"2022-01-02T17:32:09.498314Z","iopub.status.idle":"2022-01-02T17:32:36.713532Z","shell.execute_reply.started":"2022-01-02T17:32:09.498285Z","shell.execute_reply":"2022-01-02T17:32:36.712830Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Quick runtime of ~26 seconds. Let's evaluate performance with R^2, MAE, and RMSE\n\n\n# Evaluate\ny_pred = model.predict(x_test)\n\n# R^2\nprint('R^2 = ', r2_score(y_test, y_pred))\n# MAE\nprint('MAE = ', mean_absolute_error(y_test, y_pred))\n# RMSE\nprint('RMSE = ', np.sqrt(mean_absolute_error(y_test, y_pred)))","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:32:36.715356Z","iopub.execute_input":"2022-01-02T17:32:36.716203Z","iopub.status.idle":"2022-01-02T17:32:36.780847Z","shell.execute_reply.started":"2022-01-02T17:32:36.716150Z","shell.execute_reply":"2022-01-02T17:32:36.780180Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<br>","metadata":{}},{"cell_type":"markdown","source":"**Interpretation:** \n\n**R^2** indicates the percentage of the variance in the dependent variable that the independent variables explain. The results are between 0 and 1 and the closer to 1, the better. Seeing that this baseline model is at 0.98 is incredibly strong and will be difficult to improve. \n\n**MAE** measures the average magnitude of the errors in a set of predcitions. Simply put, it is the average over the test sample of the absolute differences between prediction and the actual observation. MAE is in the unit of the dependent variable, so having it at 0.03 is quite accurate. \n\n**RMSE** is the square root of the average of squared differences between the prediction and the actual observations. Like MAE, it is in the unit of the dependent variable. A RMSE of 0.17 is also quite strong. \n\n**Using Cross Validation, I attempted to improve the performance of the model.**","metadata":{}},{"cell_type":"markdown","source":"<br>\n<br>","metadata":{}},{"cell_type":"markdown","source":"## Cross Validation","metadata":{}},{"cell_type":"markdown","source":"For this model, I used an \"informed\" Grid Search Cross Validation process. By informed, I mean that I determined reasonable values to test with GridSearchCV and recorded the results. I took an iterative process to better understand how each hyper-parameter affected the model. The hypter-parameters tested and their associated values are provided below. I will not show the results of each run but will provide an example. \n\n- **Max Depth**: 3, 4, 5, 6, 7, 8, 9, 10\n- **Learning Rate**: 0.01, 0.07, 0.13, 0.19, 0.25, 0.3\n- **N Estimators**: 100, 280, 460, 640, 820, 1000\n- **Colsample_bytree**: 0.5, 0.6, 0.7, 0.8, 0.9, 1\n- **Sub Sample**: 0.5, 0.6, 0.7, 0.8, 0.9, 1\n- **Min Child Weight**: 1, 2, 3, 4, 5, 6\n- **Alpha**: 0, 1, 5, 10, 15\n- **Lambda**: 0, 1, 5, 10, 15\n- **Gamma**: 0, 1, 2, 3, 4, 5","metadata":{}},{"cell_type":"markdown","source":"**Note: I will not run the cell below on Kaggle but the code does work. Example is found on my GitHub**\n\nLink: https://github.com/seanwsullivan1/NFL-Big-Data-Bowl-2022/blob/main/NFL_Big_Data_Bowl_2022_SeanSullivan.ipynb","metadata":{}},{"cell_type":"code","source":"# Make sure you import GridSearchCV\n\n\nparams_testing = {\n        'gamma': [0, 1, 2, 3, 4, 5]\n        }\n\n\nmodel = xgb.XGBRegressor(seed=21, verbosity = 0)\n\n\ngrid_search = GridSearchCV(\n        estimator = model, \n        param_grid = params_testing, \n        scoring = 'neg_mean_squared_error', \n        n_jobs = 1, \n        cv = 10,\n        verbose = True\n        )\n\n\nstart = time.time()\ngrid_search.fit(x_train,y_train)\nprint('Runtime: ', time.time() - start)\n\n\nprint(\"Best parameters:\", grid_search.best_params_)\nprint(\"Lowest RMSE: \", (np.sqrt(-grid_search.best_score_)))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<br>","metadata":{}},{"cell_type":"markdown","source":"**Interpretation:** You can see that out of the six options, that 0 produced the lowest RMSE of 0.04. With these results in hand, I would note the run time, the results, and move forward with testing the next hyper-parameter. ","metadata":{}},{"cell_type":"markdown","source":"<br>","metadata":{}},{"cell_type":"markdown","source":"## Hyper-parameters Chosen\n\n- **Max Depth**: 10\n- **Learning Rate**: 0.19\n- **N Estimators**: 1,000\n- **Colsample_bytree**: 1\n- **Sub Sample**: 0.5\n- **Min Child Weight**: 6\n- **Alpha**: 1\n- **Lambda**: 10\n- **Gamma**: 0","metadata":{}},{"cell_type":"markdown","source":"<br>\n<br>","metadata":{}},{"cell_type":"markdown","source":"## Train Tuned Model and Evaluate Performance","metadata":{}},{"cell_type":"code","source":"# Train Model\n\n\nstart = time.time()\n\nmodel = xgb.XGBRegressor(seed = 21, max_depth = 10, learning_rate = 0.9,\n                         n_estimators = 1000, colsample_bytree = 1, sub_sample = 0.5,\n                         alpha = 1, lamba = 10, gamma = 0, min_child_weight = 6)\nmodel.fit(x_train, y_train)\nprint('Runtime: ', time.time() - start)","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:32:36.782058Z","iopub.execute_input":"2022-01-02T17:32:36.782448Z","iopub.status.idle":"2022-01-02T17:36:15.802937Z","shell.execute_reply.started":"2022-01-02T17:32:36.782415Z","shell.execute_reply":"2022-01-02T17:36:15.801979Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Generate Prediction\n\ny_pred = model.predict(x_test)","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:36:15.804986Z","iopub.execute_input":"2022-01-02T17:36:15.805233Z","iopub.status.idle":"2022-01-02T17:36:16.002248Z","shell.execute_reply.started":"2022-01-02T17:36:15.805202Z","shell.execute_reply":"2022-01-02T17:36:16.001309Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Evaluate Performance on Test Set\n\n# R^2\nprint('R^2 = ', r2_score(y_test, y_pred))\n# MAE\nprint('MAE = ', mean_absolute_error(y_test, y_pred))\n# RMSE\nprint('RMSE = ', np.sqrt(mean_absolute_error(y_test, y_pred)))","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:36:16.003873Z","iopub.execute_input":"2022-01-02T17:36:16.004169Z","iopub.status.idle":"2022-01-02T17:36:16.015030Z","shell.execute_reply.started":"2022-01-02T17:36:16.004131Z","shell.execute_reply":"2022-01-02T17:36:16.014064Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<br>","metadata":{}},{"cell_type":"markdown","source":"**Interpretation:** Interestingly enough, going through the hyper-parameter tuning process (that I chose) did not lead to a superior model to the base model. There are other methods of tuning models, but for this exercise, I will move forward with the base model and it's default settings since it's performance was superior. By superior performance, I mean that: the R^2 was higher and the MAE / RMSE were lower in the base model compared to the tuned model. ","metadata":{}},{"cell_type":"markdown","source":"<br>\n<br>","metadata":{}},{"cell_type":"markdown","source":"***","metadata":{}},{"cell_type":"markdown","source":"# Final Model and Feature Importance","metadata":{}},{"cell_type":"code","source":"# Run model with timer\n\nstart = time.time()\n\nfinal_model = xgb.XGBRegressor(seed=21)\nfinal_model.fit(x_train, y_train)\n\nprint('Runtime: ', time.time() - start)","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:36:16.016676Z","iopub.execute_input":"2022-01-02T17:36:16.017811Z","iopub.status.idle":"2022-01-02T17:36:42.527578Z","shell.execute_reply.started":"2022-01-02T17:36:16.017770Z","shell.execute_reply":"2022-01-02T17:36:42.526512Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Confirm performance\ny_pred = final_model.predict(x_test)\n\n# R^2\nprint('R^2 = ', r2_score(y_test, y_pred))\n# MAE\nprint('MAE = ', mean_absolute_error(y_test, y_pred))\n# RMSE\nprint('RMSE = ', np.sqrt(mean_absolute_error(y_test, y_pred)))","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:36:42.529081Z","iopub.execute_input":"2022-01-02T17:36:42.529367Z","iopub.status.idle":"2022-01-02T17:36:42.597998Z","shell.execute_reply.started":"2022-01-02T17:36:42.529325Z","shell.execute_reply":"2022-01-02T17:36:42.596826Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Feature Importance\nfrom xgboost import plot_importance\n\n# plot feature importance\nplot_importance(final_model, max_num_features=20)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:36:42.599841Z","iopub.execute_input":"2022-01-02T17:36:42.600175Z","iopub.status.idle":"2022-01-02T17:36:42.958513Z","shell.execute_reply.started":"2022-01-02T17:36:42.600131Z","shell.execute_reply":"2022-01-02T17:36:42.957202Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<br>","metadata":{}},{"cell_type":"markdown","source":"**Interpretation:** The results of the feature importance plot is very helpful for the interpretation of the model. Seeing that the most important features are ones that describe the state of the returner (direction, speed, x position, orientation, y position, and acceleration) makes sense. A returner's ability to gain yards certainly would appear to be dependent on their movement at that time. Additionally, seeing many kicking players' distance features show as highly important also passes the taste test. Having defenders near the returner would likely limit their ability to gain yards. It would work the other way too, where defenders being further away would make it more likely to enable a returner's ability to gain yards. I am a little surprised that the features that described the number of kicking team or returning team players within X yards of the returner did not show as more important. In conclusion, I am not totally surprised to see the results above. It has proven that the effort taken in the feature engineering stage was worth while. ","metadata":{}},{"cell_type":"markdown","source":"<br>\n<br>","metadata":{}},{"cell_type":"markdown","source":"***","metadata":{}},{"cell_type":"markdown","source":"# Run 2020 Data Through Trained Model","metadata":{}},{"cell_type":"code","source":"# Load production data\n\nproduction_data = pd.read_csv('../input/d/seanwsullivan1/nfl-big-data-bowl-2022/NFL_Big_Data_Bowl_Fall_2021_production_data.csv')","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:36:42.960869Z","iopub.execute_input":"2022-01-02T17:36:42.961279Z","iopub.status.idle":"2022-01-02T17:36:43.649601Z","shell.execute_reply.started":"2022-01-02T17:36:42.961231Z","shell.execute_reply":"2022-01-02T17:36:43.648319Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Get shape of file\n\nproduction_data.shape","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:36:43.652527Z","iopub.execute_input":"2022-01-02T17:36:43.652809Z","iopub.status.idle":"2022-01-02T17:36:43.658806Z","shell.execute_reply.started":"2022-01-02T17:36:43.652777Z","shell.execute_reply":"2022-01-02T17:36:43.657801Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# How many punt plays are being evaluated?\n\nlen(pd.unique(production_data['ultimate_id']))","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:36:43.659904Z","iopub.execute_input":"2022-01-02T17:36:43.660163Z","iopub.status.idle":"2022-01-02T17:36:43.675647Z","shell.execute_reply.started":"2022-01-02T17:36:43.660133Z","shell.execute_reply":"2022-01-02T17:36:43.674930Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**I am running 604 punt plays that were associated with a return and that did not contain a penality called in the play. If you compare the count ran with similar logic to the Plays table, it is 608 unique plays. The reason for the difference is from lableing issues noticed in the data provided and I did not override the results due to the labeling issues not aligning with my data cleaning process.**","metadata":{}},{"cell_type":"markdown","source":"<br>","metadata":{}},{"cell_type":"code","source":"# Split into numeric and categorical variables for evaluation\n\nnum_var_prod = production_data.drop(['gameID', 'playID', 'frameID', 'nflID', 'displayName', 'event', 'dis_updated', 'x_updated',\n                   'punt_received_frame', 'ultimate_id','quarter', 'advtanage', 'team', 'time_situation',\n                   'receiving_team_name', 'kicking_team_name'], 1)\n\ncat_var_prod = production_data[['playDirection', 'quarter', 'advtanage', 'team', 'time_situation']]","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:36:43.676909Z","iopub.execute_input":"2022-01-02T17:36:43.677650Z","iopub.status.idle":"2022-01-02T17:36:43.696856Z","shell.execute_reply.started":"2022-01-02T17:36:43.677607Z","shell.execute_reply":"2022-01-02T17:36:43.695838Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Evaluate Numerical Variables\n\nnum_var_prod.describe().T","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:36:43.736782Z","iopub.execute_input":"2022-01-02T17:36:43.737813Z","iopub.status.idle":"2022-01-02T17:36:44.161807Z","shell.execute_reply.started":"2022-01-02T17:36:43.737738Z","shell.execute_reply":"2022-01-02T17:36:44.160900Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Categorical Variable Exploration\n\nfor i in cat_var_prod:\n    print()\n    print(cat_var_prod[i].value_counts())\n    print('---------------------------------------------')","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:36:43.698819Z","iopub.execute_input":"2022-01-02T17:36:43.699689Z","iopub.status.idle":"2022-01-02T17:36:43.735859Z","shell.execute_reply.started":"2022-01-02T17:36:43.699642Z","shell.execute_reply":"2022-01-02T17:36:43.735235Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Check for Nulls\n\n# Numeric Variable Check\nnum_var_prod.isnull().values.any()","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:36:44.163216Z","iopub.execute_input":"2022-01-02T17:36:44.163909Z","iopub.status.idle":"2022-01-02T17:36:44.178192Z","shell.execute_reply.started":"2022-01-02T17:36:44.163698Z","shell.execute_reply":"2022-01-02T17:36:44.177466Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Categorical Variable Check\ncat_var_prod.isnull().values.any()","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:36:44.179383Z","iopub.execute_input":"2022-01-02T17:36:44.179601Z","iopub.status.idle":"2022-01-02T17:36:44.201235Z","shell.execute_reply.started":"2022-01-02T17:36:44.179574Z","shell.execute_reply":"2022-01-02T17:36:44.200235Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<br>","metadata":{}},{"cell_type":"markdown","source":"**The production data looks good. Time to run through the model.**","metadata":{}},{"cell_type":"code","source":"# Create X and dummy variables\n\nx_prod = production_data.drop(['gameID', 'playID', 'frameID', 'nflID', 'displayName', 'event', 'punt_received_frame', \n             'ultimate_id', 'dis_updated', 'x_updated', 'x_gained_dv', 'receiving_team_name', 'kicking_team_name'],1)\n\nx_prod = pd.get_dummies(x_prod)","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:36:44.203222Z","iopub.execute_input":"2022-01-02T17:36:44.204387Z","iopub.status.idle":"2022-01-02T17:36:44.260050Z","shell.execute_reply.started":"2022-01-02T17:36:44.204336Z","shell.execute_reply":"2022-01-02T17:36:44.259048Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Ensure correct columns in X\n\nlist(x_prod.columns)","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:36:44.262090Z","iopub.execute_input":"2022-01-02T17:36:44.262670Z","iopub.status.idle":"2022-01-02T17:36:44.273276Z","shell.execute_reply.started":"2022-01-02T17:36:44.262621Z","shell.execute_reply":"2022-01-02T17:36:44.272258Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Y\n\ny_prod = production_data['x_gained_dv']","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:36:44.274909Z","iopub.execute_input":"2022-01-02T17:36:44.275223Z","iopub.status.idle":"2022-01-02T17:36:44.284652Z","shell.execute_reply.started":"2022-01-02T17:36:44.275180Z","shell.execute_reply":"2022-01-02T17:36:44.283714Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Run through model\n\ny_pred_2020 = final_model.predict(x_prod)","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:36:44.285710Z","iopub.execute_input":"2022-01-02T17:36:44.286276Z","iopub.status.idle":"2022-01-02T17:36:44.386400Z","shell.execute_reply.started":"2022-01-02T17:36:44.286239Z","shell.execute_reply":"2022-01-02T17:36:44.385684Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Take results, convert to list, and attach to DF\n\nproduction_data['predicted_x_gained_dv'] = y_pred_2020.tolist()","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:36:44.387859Z","iopub.execute_input":"2022-01-02T17:36:44.392206Z","iopub.status.idle":"2022-01-02T17:36:44.405547Z","shell.execute_reply.started":"2022-01-02T17:36:44.392143Z","shell.execute_reply":"2022-01-02T17:36:44.404694Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Generate difference of the two (Return Yards Over Expected)\n\nproduction_data['x_gained_difference'] = production_data['x_gained_dv'] - production_data['predicted_x_gained_dv']","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:36:44.407326Z","iopub.execute_input":"2022-01-02T17:36:44.407930Z","iopub.status.idle":"2022-01-02T17:36:44.416697Z","shell.execute_reply.started":"2022-01-02T17:36:44.407881Z","shell.execute_reply":"2022-01-02T17:36:44.416010Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Save to CSV\n\nproduction_data.to_csv('all_2020_results.csv')","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:36:44.418195Z","iopub.execute_input":"2022-01-02T17:36:44.419061Z","iopub.status.idle":"2022-01-02T17:36:47.997107Z","shell.execute_reply.started":"2022-01-02T17:36:44.419007Z","shell.execute_reply":"2022-01-02T17:36:47.995955Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<br>\n<br>","metadata":{}},{"cell_type":"markdown","source":"***","metadata":{}},{"cell_type":"markdown","source":"# Results Analysis: So What? Why Do We Care?","metadata":{}},{"cell_type":"code","source":"production_data.head()","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:36:47.998997Z","iopub.execute_input":"2022-01-02T17:36:47.999447Z","iopub.status.idle":"2022-01-02T17:36:48.152382Z","shell.execute_reply.started":"2022-01-02T17:36:47.999387Z","shell.execute_reply":"2022-01-02T17:36:48.150664Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"players = production_data[['displayName', 'receiving_team_name', 'ultimate_id', 'x_gained_dv', 'predicted_x_gained_dv', 'x_gained_difference']]","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:36:48.154109Z","iopub.execute_input":"2022-01-02T17:36:48.154346Z","iopub.status.idle":"2022-01-02T17:36:48.176041Z","shell.execute_reply.started":"2022-01-02T17:36:48.154318Z","shell.execute_reply":"2022-01-02T17:36:48.175020Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"player_results = players.groupby(['displayName', 'receiving_team_name'])['x_gained_dv', 'predicted_x_gained_dv', 'x_gained_difference'].sum()","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:36:48.177267Z","iopub.execute_input":"2022-01-02T17:36:48.177632Z","iopub.status.idle":"2022-01-02T17:36:48.198533Z","shell.execute_reply.started":"2022-01-02T17:36:48.177600Z","shell.execute_reply":"2022-01-02T17:36:48.197435Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Sort by x_gained_difference (Return Yards Over Expected)\n\nplayer_results.sort_values(by = ['x_gained_difference'], inplace = True, ascending = False)","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:45:34.118058Z","iopub.execute_input":"2022-01-02T17:45:34.119656Z","iopub.status.idle":"2022-01-02T17:45:34.131103Z","shell.execute_reply.started":"2022-01-02T17:45:34.119556Z","shell.execute_reply":"2022-01-02T17:45:34.130267Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Get Return Count\n\nreturn_count = players.groupby(['displayName', 'receiving_team_name']).ultimate_id.nunique()","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:45:39.742686Z","iopub.execute_input":"2022-01-02T17:45:39.744013Z","iopub.status.idle":"2022-01-02T17:45:39.769403Z","shell.execute_reply.started":"2022-01-02T17:45:39.743949Z","shell.execute_reply":"2022-01-02T17:45:39.768380Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Merge Results, Rename Columns, and Look at Top 10 for Punt Return Yards Over Expected\n\nplayer_merged = pd.merge(player_results, return_count, how = 'left', on = ['displayName', 'receiving_team_name'])\n\nplayer_merged.columns = ['Return Yards Gained', 'Predicted Return Yards', 'Punt Return Yards Over Expected', 'Return Count']\n\nplayer_merged.head(10)","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:45:42.527906Z","iopub.execute_input":"2022-01-02T17:45:42.528698Z","iopub.status.idle":"2022-01-02T17:45:42.566675Z","shell.execute_reply.started":"2022-01-02T17:45:42.528661Z","shell.execute_reply":"2022-01-02T17:45:42.565853Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create Return Yards Per Return, Predicted Yards Per Return, and Punt Return Yards Over Expected Per Return\n\n# Return Yards Per Return\nplayer_merged['Return Yards Per Return'] = (player_merged['Return Yards Gained'] / player_merged['Return Count']).round(2)\n\n# Predicted Yards Per Return\nplayer_merged['Predicted Yards Per Return'] = (player_merged['Predicted Return Yards'] / player_merged['Return Count']).round(2)\n\n# Punt Return Yards Over Expected Per Return\nplayer_merged['Punt Return Yards Over Expected Per Return'] = (player_merged['Punt Return Yards Over Expected'] / player_merged['Return Count']).round(2)","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:45:46.806658Z","iopub.execute_input":"2022-01-02T17:45:46.807128Z","iopub.status.idle":"2022-01-02T17:45:46.820710Z","shell.execute_reply.started":"2022-01-02T17:45:46.807085Z","shell.execute_reply":"2022-01-02T17:45:46.819646Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"player_merged.head(10)","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:45:49.270809Z","iopub.execute_input":"2022-01-02T17:45:49.271741Z","iopub.status.idle":"2022-01-02T17:45:49.291453Z","shell.execute_reply.started":"2022-01-02T17:45:49.271679Z","shell.execute_reply":"2022-01-02T17:45:49.290621Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<br>","metadata":{}},{"cell_type":"markdown","source":"**Interpretation:**\n\nAs we saw earlier, the model was quite accurate for being able to predict how many yards a returner would gain on a punt return. So, it is reasonable to observe that Punt Return Yards Over Expected (PRYOE) is quite small given the smaller sample size of punt returns. In this case, Kalif Raymond of the Tennessee Titans had the most PRYOE +8.6 yards gained on 18 punt returns. This means, that he outgained what his total punt return yards were expected to be, by about 9 yards. \n\nLet's evaluate who has the most PRYOE per return, with at least 8 punt returns (average number of return attempts). ","metadata":{}},{"cell_type":"markdown","source":"<br>","metadata":{}},{"cell_type":"code","source":"# Filter for players with at least 8 returns. \n\nplayers_filtered = player_merged.loc[player_merged['Return Count'] >= 8]","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:45:53.065454Z","iopub.execute_input":"2022-01-02T17:45:53.066401Z","iopub.status.idle":"2022-01-02T17:45:53.071896Z","shell.execute_reply.started":"2022-01-02T17:45:53.066356Z","shell.execute_reply":"2022-01-02T17:45:53.071151Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Sort and View Results\n\nplayers_filtered.sort_values(by = ['Punt Return Yards Over Expected Per Return'], inplace = True, ascending = False)\n\nplayers_filtered","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:45:54.573337Z","iopub.execute_input":"2022-01-02T17:45:54.573776Z","iopub.status.idle":"2022-01-02T17:45:54.618739Z","shell.execute_reply.started":"2022-01-02T17:45:54.573725Z","shell.execute_reply":"2022-01-02T17:45:54.617787Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<br>","metadata":{}},{"cell_type":"markdown","source":"**Interpretation:**\n\nEvaluating the results on a per return basis, Kalif Raymond (Tennessee Titans), Andre Roberts (Buffalo Bills), and Diontae Spencer (Denver Broncos) are in a tier all to themselves nearly gaining +0.5 yards per return more than expected.  \n\nJabrill Peppers (New York Giants), Pharoh Cooper (Carolina Panthers), Marquez Callaway (New Orleans Saints), Gunner Olszewski (New England Patriots), and Jaydon Mickens (Tampa Bay Buccaneers) were in the next tier gaining nearly +0.3 yards per return more than expected. \n\nThe players with the worst PRYOE per play were Christian Kirk (Arizona Cardinals), Mecole Hardman (Kansas City Chiefs), and Jakeem Grant (Miami Dolphins). \n\n**I will perform a deep dive on what made Kalif Raymond so successful at outperforming his punt returns and what made Jakeem Grant so poor at his. Grant is especially interesting because he ranks first for total yards gained in these results, but is the third worst returner on a PRYOE per play lens.**","metadata":{}},{"cell_type":"markdown","source":"<br>\n<br>","metadata":{}},{"cell_type":"markdown","source":"## Case Study: Kalif Raymond and Jakeem Grant","metadata":{}},{"cell_type":"code","source":"# Evaluate Kalif Raymond's PRYOE\n\nproduction_data[production_data.displayName == 'Kalif Raymond'].x_gained_difference.describe()","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:45:58.237065Z","iopub.execute_input":"2022-01-02T17:45:58.237970Z","iopub.status.idle":"2022-01-02T17:45:58.257722Z","shell.execute_reply.started":"2022-01-02T17:45:58.237896Z","shell.execute_reply":"2022-01-02T17:45:58.256737Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Visualize Kalif Raymond's Returns\n\nplt.hist(production_data[production_data.displayName == 'Kalif Raymond'].x_gained_difference, bins=20)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:46:00.724354Z","iopub.execute_input":"2022-01-02T17:46:00.724944Z","iopub.status.idle":"2022-01-02T17:46:00.996059Z","shell.execute_reply.started":"2022-01-02T17:46:00.724902Z","shell.execute_reply":"2022-01-02T17:46:00.995131Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Evaluate Actual Yards Gained\n\nproduction_data[production_data.displayName == 'Kalif Raymond'].groupby(['ultimate_id'])['x_gained_dv'].sum().describe()","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:46:03.420642Z","iopub.execute_input":"2022-01-02T17:46:03.421540Z","iopub.status.idle":"2022-01-02T17:46:03.439951Z","shell.execute_reply.started":"2022-01-02T17:46:03.421480Z","shell.execute_reply":"2022-01-02T17:46:03.438983Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Evaluate Actual Yards Gained Distribution\n\nplt.hist(production_data[production_data.displayName == 'Kalif Raymond'].groupby(['ultimate_id'])['x_gained_dv'].sum())\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:50:43.951999Z","iopub.execute_input":"2022-01-02T17:50:43.952646Z","iopub.status.idle":"2022-01-02T17:50:44.124645Z","shell.execute_reply.started":"2022-01-02T17:50:43.952572Z","shell.execute_reply":"2022-01-02T17:50:44.123780Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<br>","metadata":{}},{"cell_type":"markdown","source":"**Look at the same thing, but for Jakeem Grant**","metadata":{}},{"cell_type":"code","source":"# Evaluate Jakeem Grant's PRYOE\n\nproduction_data[production_data.displayName == 'Jakeem Grant'].x_gained_difference.describe()","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:46:05.553113Z","iopub.execute_input":"2022-01-02T17:46:05.553706Z","iopub.status.idle":"2022-01-02T17:46:05.572232Z","shell.execute_reply.started":"2022-01-02T17:46:05.553629Z","shell.execute_reply":"2022-01-02T17:46:05.571404Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Visualize Jakeem Grant's Returns\n\nplt.hist(production_data[production_data.displayName == 'Jakeem Grant'].x_gained_difference, bins=20)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:46:07.577331Z","iopub.execute_input":"2022-01-02T17:46:07.577745Z","iopub.status.idle":"2022-01-02T17:46:07.868682Z","shell.execute_reply.started":"2022-01-02T17:46:07.577714Z","shell.execute_reply":"2022-01-02T17:46:07.867639Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Evaluate Actual Yards Gained\n\nproduction_data[production_data.displayName == 'Jakeem Grant'].groupby(['ultimate_id'])['x_gained_dv'].sum().describe()","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:46:09.754307Z","iopub.execute_input":"2022-01-02T17:46:09.754589Z","iopub.status.idle":"2022-01-02T17:46:09.775472Z","shell.execute_reply.started":"2022-01-02T17:46:09.754551Z","shell.execute_reply":"2022-01-02T17:46:09.773762Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Evaluate Actual Yards Gained Distribution\n\nplt.hist(production_data[production_data.displayName == 'Jakeem Grant'].groupby(['ultimate_id'])['x_gained_dv'].sum())\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:46:12.230706Z","iopub.execute_input":"2022-01-02T17:46:12.231026Z","iopub.status.idle":"2022-01-02T17:46:12.442684Z","shell.execute_reply.started":"2022-01-02T17:46:12.230994Z","shell.execute_reply":"2022-01-02T17:46:12.441387Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<br>","metadata":{}},{"cell_type":"markdown","source":"**Interpretation**: \n\nA quick look at punt returns for Raymond and Grant show that Grant, on average, gained more return yards than Raymond. But, this is slightly skewed from an 86 yard return (for a touchdown) by Grant. If you evaluate the 75th percentile of their return yard distribution, Raymond slightly out-gained Grant. \n\nWe already know that Raymond was the best returner on a PRYOE per Return basis and that Grant was one of the worst. So what made one better than the other? Was it a factor of their teammates? Was it individual ability? I dove into the features of the model to try to understand what dynamics were at play. For this, I performed the data crunching in BigQuery but will provide the file and interpretation below. ","metadata":{}},{"cell_type":"markdown","source":"<br>","metadata":{}},{"cell_type":"code","source":"# Load CSV\n\ncase_study = pd.read_csv('../input/d/seanwsullivan1/nfl-big-data-bowl-2022/raymond_and_grant_results.csv')","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:46:15.228077Z","iopub.execute_input":"2022-01-02T17:46:15.228384Z","iopub.status.idle":"2022-01-02T17:46:15.248638Z","shell.execute_reply.started":"2022-01-02T17:46:15.228350Z","shell.execute_reply":"2022-01-02T17:46:15.247989Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Filter Results\n\ncs_filtered = case_study[['displayName', 'total_yards_gained_rank', 'total_return_yards_over_expected_ranked', 'rec_to_kick_5YRD_AVG__Diff_Rank', 'kb1_frame_per_return_rank', \n                          'return_yards_per_play_rank', 'predicted_return_yards_per_play_rank', 'pryoe_per_play_rank']]","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:46:17.198184Z","iopub.execute_input":"2022-01-02T17:46:17.198486Z","iopub.status.idle":"2022-01-02T17:46:17.204835Z","shell.execute_reply.started":"2022-01-02T17:46:17.198453Z","shell.execute_reply":"2022-01-02T17:46:17.203834Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cs_filtered","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:46:19.278740Z","iopub.execute_input":"2022-01-02T17:46:19.279093Z","iopub.status.idle":"2022-01-02T17:46:19.292572Z","shell.execute_reply.started":"2022-01-02T17:46:19.279053Z","shell.execute_reply":"2022-01-02T17:46:19.291617Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<br>","metadata":{}},{"cell_type":"markdown","source":"**Interpretation**\n\nYou may notice some unfamiliar metric rankings here, but I will explain. As you already know, Jakeem Grant was one of the best returners in terms of total return yards gained and Kalif Raymond was one of the better, but not the best. We also already know that Raymond was the best punt returner based on PRYOE and Grant was one of the worst. \n\n\"Rec_to_Kick_5YRD_AVG_Diff_Rank\" is the difference of the average returning team member count within 5 yards of the returner and the average kicking team member count within 5 yards, but ranked amongst the players. Basically, on average (on a frame level), how many more returner teammates are within 5 yards of the returner than kicking team members. The higher the rank, the more, on average, teammates within 5 yards than opponents. Which, would suggest despite the crowdedness, that the returner had teammates providing blocking/resitance to the kicking team. In this case, Raymond was 30th out of 34 (filtered out to only include returners with 8 or more returns) and Grant was 7th out of 34. This tells us that Raymond typically had more defenders than teammates within 5 yards of him and Grant had the opposite. So, this would suggest that Grant had a better blocking situation whereas Raymond did not. This may hint at Raymond having superior returning \"vision\", but this can not be proven with this information. \n\n\"KB1_Frame_Per_Return\" is counting the number of frames, per play, with more than one kicking team player within 5 yards of the returner, but ranked amongst the players. This is providing insight to how much pressure a returner faces on a typical play. This is slightly inflated if a returner is known to \"run around\" quite a bit and accumulate more frames within a play, so take this with a grain of salt. But, with that said, Grant ranks 5th highest for frames with defenders within 5 yards per play and Raymond ranks 15th. \n\nThese two measurements present conflicting insight on a high level, but provide insight nonetheless. Grant typically had more teammtes than defenders within 5 yards than Raymond, but Grant typically had more defenders, on a frame level, within 5 yards of him. This may be an indicator into why Grant underperformed so badly when evaluated with PRYOE. On the other hand, Raymond typically had more defenders within 5 yards of him than teammates, but he was middle fo the pack regarding the number of frames with defenders within 5 yards of him. Given that Raymond tended to have a tougher \"situation\" and that he outperformed his expecation, it is reasonable to ponder what his punt returning performance would be with a better blocking scheme.  ","metadata":{}},{"cell_type":"markdown","source":"<br>","metadata":{}},{"cell_type":"markdown","source":"## Visualize Jakeem Grant Punt Returns","metadata":{}},{"cell_type":"markdown","source":"In the visualizations below, I animated two punt returns by Jakeem Grant. One was a punt return for a touchdown and the other was a return with negative PRYOE accumulated in the play. This is meant to help show what a play with positive associated PRYOE and negative PRYOE look like. The size of the returner's bubble is reflecting the predicted yards gained in a frame. I ended up scaling the metric to ensure that the players would be visible within the animation. \n\nThe code below was modified from code posted by Cube Root at https://www.kaggle.com/threecifanggen/replay-the-game-using-plotly-to-animate-the-game. ","metadata":{}},{"cell_type":"markdown","source":"<br>","metadata":{}},{"cell_type":"code","source":"import plotly.express as px\nfrom plotly.offline import init_notebook_mode\nimport plotly.graph_objects as go\ninit_notebook_mode()\nimport ipywidgets as widgets","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:46:22.281779Z","iopub.execute_input":"2022-01-02T17:46:22.282083Z","iopub.status.idle":"2022-01-02T17:46:23.868167Z","shell.execute_reply.started":"2022-01-02T17:46:22.282051Z","shell.execute_reply":"2022-01-02T17:46:23.867049Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Jakeem Grant Punt Return for TD (GameID: 2020110106, PlayID: 1473) ","metadata":{}},{"cell_type":"code","source":"tracking_20_df = pd.read_csv('../input/d/seanwsullivan1/nfl-big-data-bowl-2022/2020_11_01_06_JakeemGrant_TD_Return.csv')","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:46:25.130540Z","iopub.execute_input":"2022-01-02T17:46:25.130859Z","iopub.status.idle":"2022-01-02T17:46:25.165830Z","shell.execute_reply.started":"2022-01-02T17:46:25.130827Z","shell.execute_reply":"2022-01-02T17:46:25.165127Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_play_in_game(gameId, speed=50):\n    temp_tracking_df = tracking_20_df[tracking_20_df['gameId'] == gameId]\n    for playId in temp_tracking_df['playId'].dropna().unique():\n        temp_tracking_query = (tracking_20_df['gameId'] == gameId) & (tracking_20_df['playId'] == playId)\n        temp_tracking_df = (\n            tracking_20_df[temp_tracking_query][['x', 'y', 'frameId', 'nflId', 'team', 'displayName', 'predicted_yards_gained']]\n            .fillna(0.)\n            .sort_values(['team', 'frameId'])\n        )\n        color_discrete_map = {'home': 'rgb(0,142,151)', 'away': 'rgb(255,163,0)', 'football': 'rgb(49,29,0)'}\n        fig = px.scatter(\n            temp_tracking_df,\n            x='x',\n            y='y',\n            animation_frame='frameId',\n            color='team',\n            color_discrete_map=color_discrete_map,\n            animation_group=\"nflId\",\n            hover_name=\"displayName\",\n            size = 'predicted_yards_gained'\n        )\n        fig.update_traces(marker=dict(size=12,\n                                      line=dict(width=2,\n                                                color='DarkSlateGrey')),\n                          selector=dict(mode='markers'))\n\n        ## Drawing the Ground\n        for x in range(0, 130, 10):\n            fig.add_trace(go.Scatter(x=[x, x], y=[0, 53.3], mode='lines', showlegend=False, line=dict(color=\"#333333\")))\n        fig.add_trace(go.Scatter(x=[0, 120], y=[53.3, 53.3], mode='lines', showlegend=False, line=dict(color=\"#333333\")))\n        fig.add_trace(go.Scatter(x=[0, 120], y=[0, 0], mode='lines', showlegend=False, line=dict(color=\"#333333\")))\n        fig.update_layout(\n            autosize=False,\n            width=1100,\n            height=600,\n            title=f'Jakeem Grant (MIA) vs LAR: Punt Return for 86 Yards (PRYOE: +5.24)',\n        )\n        fig.layout.updatemenus[0].buttons[0].args[1][\"frame\"][\"duration\"] = speed\n        fig.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:46:30.560100Z","iopub.execute_input":"2022-01-02T17:46:30.560379Z","iopub.status.idle":"2022-01-02T17:46:30.575385Z","shell.execute_reply.started":"2022-01-02T17:46:30.560350Z","shell.execute_reply":"2022-01-02T17:46:30.574664Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gameId = 2020110106\nplot_play_in_game(gameId)","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:46:33.158907Z","iopub.execute_input":"2022-01-02T17:46:33.159470Z","iopub.status.idle":"2022-01-02T17:46:38.231614Z","shell.execute_reply.started":"2022-01-02T17:46:33.159432Z","shell.execute_reply":"2022-01-02T17:46:38.226842Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Filter results for gameID = 2020110106 and playID = 1473\n# Note, only frames between punt received and touchdown\n\njg_td = production_data.loc[(production_data['gameID'] == 2020110106) & (production_data['playID'] == 1473)]","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:46:43.831856Z","iopub.execute_input":"2022-01-02T17:46:43.832544Z","iopub.status.idle":"2022-01-02T17:46:43.839200Z","shell.execute_reply.started":"2022-01-02T17:46:43.832496Z","shell.execute_reply":"2022-01-02T17:46:43.838290Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Line Chart\n\njg_td.plot(x = 'frameID', y = 'x_gained_difference', kind = 'line')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:46:45.772244Z","iopub.execute_input":"2022-01-02T17:46:45.772551Z","iopub.status.idle":"2022-01-02T17:46:46.012280Z","shell.execute_reply.started":"2022-01-02T17:46:45.772521Z","shell.execute_reply":"2022-01-02T17:46:46.011248Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This line chart shows the PRYOE for the animated play above. You can see that the model is expected Grant to gain more yards after initially catching the punt. Frames 100 - 120 are where it gets interesting. According to the model, this is where Grant is really outperforming expectation. Near frame 140 is where Grant made the punter and another defender miss and from there, it was off the the races. ","metadata":{}},{"cell_type":"markdown","source":"<br>","metadata":{}},{"cell_type":"markdown","source":"### Jakeem Grant Punt Return with Poor PRYOE (GameID: 2020092004, PlayID: 3071)","metadata":{}},{"cell_type":"code","source":"tracking_20_df = pd.read_csv('../input/d/seanwsullivan1/nfl-big-data-bowl-2022/2020_09_20_04_JakeemGrant_Poor_Return.csv')","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:47:26.571433Z","iopub.execute_input":"2022-01-02T17:47:26.571975Z","iopub.status.idle":"2022-01-02T17:47:26.604509Z","shell.execute_reply.started":"2022-01-02T17:47:26.571927Z","shell.execute_reply":"2022-01-02T17:47:26.603835Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_play_in_game(gameId, speed=50):\n    temp_tracking_df = tracking_20_df[tracking_20_df['gameId'] == gameId]\n    for playId in temp_tracking_df['playId'].dropna().unique():\n        temp_tracking_query = (tracking_20_df['gameId'] == gameId) & (tracking_20_df['playId'] == playId)\n        temp_tracking_df = (\n            tracking_20_df[temp_tracking_query][['x', 'y', 'frameId', 'nflId', 'team', 'displayName', 'predicted_yards_gained']]\n            .fillna(0.)\n            .sort_values(['team', 'frameId'])\n        )\n        color_discrete_map = {'home': 'rgb(0,142,151)', 'away': 'rgb(198,12,48)', 'football': 'rgb(49,29,0)'}\n        fig = px.scatter(\n            temp_tracking_df,\n            x='x',\n            y='y',\n            animation_frame='frameId',\n            color='team',\n            color_discrete_map=color_discrete_map,\n            animation_group=\"nflId\",\n            hover_name=\"displayName\",\n            size = 'predicted_yards_gained'\n        )\n        fig.update_traces(marker=dict(size=12,\n                                      line=dict(width=2,\n                                                color='DarkSlateGrey')),\n                          selector=dict(mode='markers'))\n\n        ## Drawing the Ground\n        for x in range(0, 130, 10):\n            fig.add_trace(go.Scatter(x=[x, x], y=[0, 53.3], mode='lines', showlegend=False, line=dict(color=\"#333333\")))\n        fig.add_trace(go.Scatter(x=[0, 120], y=[53.3, 53.3], mode='lines', showlegend=False, line=dict(color=\"#333333\")))\n        fig.add_trace(go.Scatter(x=[0, 120], y=[0, 0], mode='lines', showlegend=False, line=dict(color=\"#333333\")))\n        fig.update_layout(\n            autosize=False,\n            width=1100,\n            height=600,\n            title=f'Jakeem Grant (MIA) vs BUF: Punt Return for 10 Yards (PRYOE: -10.79)',\n        )\n        fig.layout.updatemenus[0].buttons[0].args[1][\"frame\"][\"duration\"] = speed\n        fig.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:47:16.311530Z","iopub.execute_input":"2022-01-02T17:47:16.312045Z","iopub.status.idle":"2022-01-02T17:47:16.327067Z","shell.execute_reply.started":"2022-01-02T17:47:16.311996Z","shell.execute_reply":"2022-01-02T17:47:16.326202Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gameId = 2020092004\nplot_play_in_game(gameId)","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:47:29.592480Z","iopub.execute_input":"2022-01-02T17:47:29.593326Z","iopub.status.idle":"2022-01-02T17:47:32.638455Z","shell.execute_reply.started":"2022-01-02T17:47:29.593286Z","shell.execute_reply":"2022-01-02T17:47:32.637450Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Filter results for gameID = 2020092004 and playID = 3071\n# Note, only frames between punt received and touchdown\n\njg_poor = production_data.loc[(production_data['gameID'] == 2020092004) & (production_data['playID'] == 3071)]","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:47:37.213721Z","iopub.execute_input":"2022-01-02T17:47:37.214036Z","iopub.status.idle":"2022-01-02T17:47:37.220915Z","shell.execute_reply.started":"2022-01-02T17:47:37.214004Z","shell.execute_reply":"2022-01-02T17:47:37.219773Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Line Chart\n\njg_poor.plot(x = 'frameID', y = 'x_gained_difference', kind = 'line')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:47:39.483436Z","iopub.execute_input":"2022-01-02T17:47:39.484073Z","iopub.status.idle":"2022-01-02T17:47:39.662536Z","shell.execute_reply.started":"2022-01-02T17:47:39.484038Z","shell.execute_reply":"2022-01-02T17:47:39.661601Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This line chart shows the PRYOE for the animated play above. You can see that the model is expecting Grant to gain more yards than he did for most of this play. His performance really cratered in the range of frame 100 - 130. Looking at the animated chart, this is where Grant begins to run more to the left (loosing yards) and down on the y axis. It is interesting to see the model pick up a better PRYOE around frame 120 when Grant made a defender miss a tackle. ","metadata":{}},{"cell_type":"markdown","source":"<br>\n<br>","metadata":{}},{"cell_type":"markdown","source":"## Team level Rankings and Results","metadata":{}},{"cell_type":"markdown","source":"To evaluate PRYOE on a team level, I aggregated the results for teams for their returning plays and their punting plays. The returning results mostly align with the player level - which makes sense given the low sample size of punts that are returned. The punt defending results are interesting and begin to hint at units that are superior at limiting strong returns.  \n\nNote: I performed the roll-ups in BQ in order to de-clutter the notebook now that we are towards the end. ","metadata":{}},{"cell_type":"markdown","source":"<br>","metadata":{}},{"cell_type":"code","source":"# Load Team level Returning Results. Note: Ranked Highest to Lowest for PRYOE per play\n\nreceiving_team_results = pd.read_csv('../input/d/seanwsullivan1/nfl-big-data-bowl-2022/receiving_team_rankings.csv')\n\nreceiving_team_results[['receiving_team_name', 'total_yards_gained_rank', 'total_return_yards_over_expected_ranked', 'total_return_yards_over_expected__per_play_ranked']]","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:47:44.298140Z","iopub.execute_input":"2022-01-02T17:47:44.298416Z","iopub.status.idle":"2022-01-02T17:47:44.323719Z","shell.execute_reply.started":"2022-01-02T17:47:44.298387Z","shell.execute_reply":"2022-01-02T17:47:44.322544Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Load Team level Punt Defending Results. Note: Ranked Lowest to Highest for PRYOE per play\n\nkicking_team_results = pd.read_csv('../input/d/seanwsullivan1/nfl-big-data-bowl-2022/kicking_team_rankings.csv')\n\nkicking_team_results[['kicking_team_name', 'total_yards_gained_rank', 'total_return_yards_over_expected_ranked', 'total_return_yards_over_expected__per_play_ranked']]","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:47:47.110494Z","iopub.execute_input":"2022-01-02T17:47:47.110992Z","iopub.status.idle":"2022-01-02T17:47:47.136411Z","shell.execute_reply.started":"2022-01-02T17:47:47.110950Z","shell.execute_reply":"2022-01-02T17:47:47.135829Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Let's visualize the Speical Teams Unit performance (returning and punting) for PRYOE by Team\n\nspecial_teams_results = pd.read_csv('../input/d/seanwsullivan1/nfl-big-data-bowl-2022/speical_teams_comparison.csv')\n\nspecial_teams_results.head()","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:47:50.967493Z","iopub.execute_input":"2022-01-02T17:47:50.968173Z","iopub.status.idle":"2022-01-02T17:47:51.023397Z","shell.execute_reply.started":"2022-01-02T17:47:50.968104Z","shell.execute_reply":"2022-01-02T17:47:51.022305Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Scatter Plot Visualization\n\nfig = px.scatter(special_teams_results, x='PRYOE_Returning_Per_Play', y='PYROE_Kicking_Per_Play', text='Team')\nfig.update_traces(textposition='top center')\nfig.update_layout(title_text='Special Teams Unit Performance', title_x=0.5)\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-02T17:47:53.010269Z","iopub.execute_input":"2022-01-02T17:47:53.010537Z","iopub.status.idle":"2022-01-02T17:47:53.290372Z","shell.execute_reply.started":"2022-01-02T17:47:53.010509Z","shell.execute_reply":"2022-01-02T17:47:53.289722Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Interpretation:**\n\nEvaluating the scatter plot, we should use the 0,0 coordinates to base ourselves. Anything to the right of the 0 on the x axis indicates a positive PRYOE per play as the returning team (good). Anything to the left indicates a negative value (not ideal). Any value above the 0 on the y axis indicates a positive PYROE per play as the kicking team (not ideal) and anything below it indicates a negative value (good). \n\nWhere teams would want to be is the bottom right hand quadrant (positive PRYOE as a returning team and negative PRYOE as a kicking team). Using this criteria, **the Buffalo Bills** are one of the strongest punting units. Their PRYOE per play as a returning unit is slightly above 0.4 yards and as a kicking team, it is about -1.5 yards. Other teams with strong unit performance include: **New England Patriots, Indianapolis Colts, and the Philadelphia Eagles.**","metadata":{}},{"cell_type":"markdown","source":"<br>\n<br>","metadata":{}},{"cell_type":"markdown","source":"***","metadata":{}},{"cell_type":"markdown","source":"# Value of PRYOE Discussion","metadata":{}},{"cell_type":"markdown","source":"Now that the model has been explained and the results discussed, I think it is important to discuss the value of this metric. Should this metric take over in place of actual return yards gained? Probably not. But, I do think that it possess value for a few different areas of relevance. \n\n**Value Proposition 1: Post Performance Evaluation**\nPRYOE provides insight into how a returner (or kicking unit) performed relative to how they would be expected to perform given the situation of the punt return. As shown in the above section, insight can be given into if a returner was due more credit for their over performance or if their blocking unit provided a beneficial boost to the return. It is a method that begins to get to the individual contribution that the returner has on a play. \n\n\n**Value Proposition 2: In Game Strategy**\nTeams would be able to layer in this metric to help guide in game strategic decision making. For example, perhaps the Chicago Bears are playing the Buffalo Bills and know that the Bills have one of the best units on a kicking defense PRYOE per play basis. Knowing this, the Special Teams Coordinator may encourage their returner to take a fair catch or allow a punt to drop in order to maximize starting position for the ensuing drive. \n\n\nI also think that this PRYOE metric is a piece to a decision optimization problem: what is the optimal choice for a punt return (return, fair catch, let bounce)? By creating a very robust RYOE model, this tackles the first part of this decision making equation. Models would need to be developed to estimate how a punt would bounce and then provide the expected return yards, fair catch location, and bounced punt location on a given punt, in order to determine if a decision was optimal. I am aware of a few BDB projects doing this and am excited to see their finished products. ","metadata":{}},{"cell_type":"markdown","source":"<br>","metadata":{}},{"cell_type":"markdown","source":"## Limitations and Next Steps\n\nAs I mentioned in the value discussion above, there is a limitation to the PRYOE metric due to how narrow its scope is. \n\nIn terms of a next step, I think it would be worth expanding some of the player tracking features. Perhaps including features that provide information on how close players are to each other (and not just the returner) would aid in the accuracy of the model. This sort of robust feature set could also be a launching point for quantifying impact of individual players' on a given play. For example, perhaps we'd be able to quantify a kicking team defender's impact on limiting return yards or we could quantify a player on the returning team's impact on blocking and creating more return yards. However, a discussion would need to be had on how much impact this would truly bring to a team and the sports analyics community. Essentially, is the juice worth the squeeze given the number of punt returns that happen within a given season? Reagrdless, it would set the wheels in motion for similar analyses that could be performed on other actions within a football game. ","metadata":{}},{"cell_type":"markdown","source":"<br>\n<br>","metadata":{}},{"cell_type":"markdown","source":"***","metadata":{}},{"cell_type":"markdown","source":"# Acknowledgements\n\nSpcial thank you to the following people for offering direct help in serving as a sounding board and/or encouraging me to complete the project:\n\n- **My girlfriend, Sarah**\n- **Ben Draus**\n- **Tej Seth**","metadata":{}}]}