{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\nfrom IPython.display import Image, display\nimport pickle\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom scipy import stats","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-06T19:22:14.226708Z","iopub.execute_input":"2022-01-06T19:22:14.227556Z","iopub.status.idle":"2022-01-06T19:22:15.161114Z","shell.execute_reply.started":"2022-01-06T19:22:14.227443Z","shell.execute_reply":"2022-01-06T19:22:15.160196Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Introduction","metadata":{}},{"cell_type":"markdown","source":"This notebook is my submission for the NFL Big Data Bowl 2022. \n\nThe recently (2019) developed temporal fusion transformer (TFT) uses a multidimensional time series as input to forecast the multidimensional future. In this project, it predicts the next coordinates, speed, acceleration, direction, orientation, and distance of all 22 players in kickoff returns given past frames as input. By iteratively using the network's predictions as future input, one can continuously predict the next frame and simulate kickoff returns until the returner is tackled, runs out of bounds, or scores a touchdown.\n\nUsing a trained TFT, I ranked players based on hypothetical scenarios. A ranking system based solely on historical data is unnormalized in that teams and players face different teams and lineups in a given year increasing the factor of luck into rankings. My rankings are computed based on testing teams and players against the same set of conditions, i.e. the lineup of the most frequent player at each position, and could potentially provide a decomposition based on returners' out-of-system performances when returners are substituted into other teams' special teams lineups.\n\nThe tracking data is processed to a set of variables, some varying during each play (e.g. position) and some staying static, the same, (e.g. returner's weight), that represent a time instance. The aforementioned variables the network predicts are also considered as time varying input variables and are added to the input to the network when they precede the current time index. The static variables are as follows for each of the 22 players:\n\n1. Weight\n2. Height\n3. Non-special teams position\n4. 40 yd dash\n5. Vertical jump\n6. Bench press\n7. Broad jump\n8. 3Cone\n9. Shuttle run\n10. Draft pick number\n11. Returner index\n12. Kickoff team name\n13. Returning team name\n14. Play direction\n15. Kicking team home or away.\n\nSome additional pre-processing steps were applied. The variables were transformed so that the kickoff occurs on the right-hand side of the field to the left-hand side endzone. If combine data was not available for a player, the average score for that position was assigned to the player. Before all variables are input to the network, the players' names are sorted in increasing y-direction for the kicking team and increasing x-direction, then increasing y-direction for the receiving team. Data from 2018, 2019, and 2020 input into the trainer while the last 64 plays in 2020 were withheld, from which I conducted evaluations.","metadata":{}},{"cell_type":"markdown","source":"# Correlated Results Between Real and Simulated Return Yardage","metadata":{}},{"cell_type":"code","source":"file = open(\"/kaggle/input/nfl2022-input-data/data/evaluations/yards_sim_vs_real.pkl\",'rb')\nsim_real_data = pd.DataFrame( pickle.load(file) )\nfile.close()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-06T19:22:15.162667Z","iopub.execute_input":"2022-01-06T19:22:15.162903Z","iopub.status.idle":"2022-01-06T19:22:15.179505Z","shell.execute_reply.started":"2022-01-06T19:22:15.162876Z","shell.execute_reply":"2022-01-06T19:22:15.178835Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.set_theme(color_codes=True)\nslope, intercept, r_value, p_value, std_err = stats.linregress(sim_real_data['simulated'],sim_real_data['real'])\n\nax = sns.regplot(x=\"simulated\", y=\"real\", data=sim_real_data, line_kws={'label':\"real={0:.2f}sim+{1:.2f}\".format(slope,intercept)})\n\nax.legend()\nplt.title('Simulated vs. Actual Return Yardage, r^2 = {0:.2f}, p = {1:.3f}'.format(r_value, p_value))\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-06T19:22:15.180665Z","iopub.execute_input":"2022-01-06T19:22:15.181342Z","iopub.status.idle":"2022-01-06T19:22:15.586647Z","shell.execute_reply.started":"2022-01-06T19:22:15.181310Z","shell.execute_reply":"2022-01-06T19:22:15.585788Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Figure 1 depicts a regression between the simulated and real return yardages for 64 plays which the neural network has not previously seen. There exists a positive, significant correlation of 0.38 between these values. There seems to exist a systematic bias, which can adjust for by using the net differences between simulated and actual to rank the players.","metadata":{}},{"cell_type":"markdown","source":"# Visual Comparison of Real and Simulated Kickoff Return","metadata":{}},{"cell_type":"markdown","source":"Below, are Figures 2 and 3 which a real play as well as the play simulated using just the first four frames. In the simulation, all players and the teams playing are different from that in the real. ","metadata":{}},{"cell_type":"markdown","source":"**Real Play**","metadata":{}},{"cell_type":"code","source":"Image(url = \"https://github.com/mattonics/NFL-BigData-Bowl-2022/blob/main/GIFs/anim_exam.gif?raw=true\", width = 1000)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-06T19:22:15.588389Z","iopub.execute_input":"2022-01-06T19:22:15.588701Z","iopub.status.idle":"2022-01-06T19:22:15.594026Z","shell.execute_reply.started":"2022-01-06T19:22:15.588670Z","shell.execute_reply":"2022-01-06T19:22:15.593417Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Simulated Play**","metadata":{}},{"cell_type":"code","source":"Image(url = \"https://github.com/mattonics/NFL-BigData-Bowl-2022/blob/main/GIFs/anim_exam_sim.gif?raw=true\", width = 1000)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-06T19:22:15.595089Z","iopub.execute_input":"2022-01-06T19:22:15.595431Z","iopub.status.idle":"2022-01-06T19:22:15.608117Z","shell.execute_reply.started":"2022-01-06T19:22:15.595403Z","shell.execute_reply":"2022-01-06T19:22:15.607243Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Performance of Rankings\n\nReturners with their most frequent players at each blocking position were simulated against each team's kicking lineup with the most frequently appearing players at each position. 32 returners were simulated 32 times each on the same play starting from the frame the returner received the ball. Below, Figure 4 shows the ranking of how far the returner went on average when facing all 32 teams vs. the ranking of the returner's team average kickoff return yards in the 2020 season taken from NFL.com. The subplot on the right shows that although there does not exist a correlation on the whole, the data restricted to the top half of simulation yardage exhibits positive Spearman correlation of 0.45 with p-value of 0.08. The top ranked returner in simulation is Andre Roberts who was voted to the Pro Bowl in that season and belonged to the team with the highest average kickoff return yards in 2020, the Buffalo Bills.","metadata":{}},{"cell_type":"code","source":"file = open(\"/kaggle/input/nfl2022-input-data/1-4-22 Upload/figure_data_ave_returns_simulated.pkl\",'rb')\nave_returns_simulated = pickle.load(file)\nfile.close()\n\nfile = open(\"/kaggle/input/nfl2022-input-data/1-4-22 Upload/figure_data_ave_returns_team_ranked.pkl\",'rb')\nave_returns_team_ranked = pickle.load(file)\nfile.close()\n\nfig, axs = plt.subplots(1,2)\nfig.set_size_inches(10.5, 5.75)\nfig.tight_layout(pad=3.0)\nfig.suptitle('Simulated vs. Historical Ave Return Yardage Ranking')\naxs[1].scatter(ave_returns_simulated[ave_returns_simulated <= 16], ave_returns_team_ranked[ave_returns_simulated <= 16])\naxs[1].set_aspect('equal')\nplt.xlim(0,33)\nplt.ylim(0,33)\naxs[1].set_xlabel('Simulated Rank')\naxs[1].set_ylabel('Actual Rank')\naxs[1].set_title('Top Half of Simulated Rankings')\naxs[0].scatter(ave_returns_simulated, ave_returns_team_ranked)\naxs[0].set_aspect('equal')\nplt.xlim(0,33)\nplt.ylim(0,33)\naxs[0].set_xlabel('Simulated Rank')\naxs[0].set_ylabel('Actual Rank')\naxs[0].set_title('All Players')\nplt.show()\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-06T19:22:15.609421Z","iopub.execute_input":"2022-01-06T19:22:15.609960Z","iopub.status.idle":"2022-01-06T19:22:16.104245Z","shell.execute_reply.started":"2022-01-06T19:22:15.609916Z","shell.execute_reply":"2022-01-06T19:22:16.103427Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The full rankings of kickoff returners and their associated teams are shown below in Table 1.","metadata":{}},{"cell_type":"code","source":"file = open(\"/kaggle/input/nfl2022-input-data/figures/player_system_rankings.txt\",'rb')\nrankings = file.read().decode('UTF-8')\nfile.close()\nprint(rankings)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-06T19:22:16.105497Z","iopub.execute_input":"2022-01-06T19:22:16.106373Z","iopub.status.idle":"2022-01-06T19:22:16.119212Z","shell.execute_reply.started":"2022-01-06T19:22:16.106324Z","shell.execute_reply":"2022-01-06T19:22:16.118185Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Explainability of AI","metadata":{}},{"cell_type":"markdown","source":"As the TFT neural network used for simulation utilizes an attention-based architecture it can be easily explainable. Below in Figure 5, we can see that only the preceding two frames of a single instant are much more useful compared to the previous eight to predict the next behavior of all the players on the field. A higher attention score means a greater weight is applied to that variable.","metadata":{}},{"cell_type":"code","source":"Image(\"/kaggle/input/nfl2022-input-data/1-4-22 Upload 2/attention.png\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-06T19:22:16.120506Z","iopub.execute_input":"2022-01-06T19:22:16.121096Z","iopub.status.idle":"2022-01-06T19:22:16.134705Z","shell.execute_reply.started":"2022-01-06T19:22:16.121052Z","shell.execute_reply":"2022-01-06T19:22:16.134083Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Implications","metadata":{}},{"cell_type":"markdown","source":"1. As this prediction model does not require any historical data, NFL teams could use it to rank players from the NFL combine to assess how well they fit in with their set of blockers and their team prior to the season starting.\n2. This model provides an assessment of how well the kick returners would perform outside of their team and could be used to scout good fits for trades or free agent signings.\n3. This model could hopefully be extended to punt returns, rushing plays, passing plays, etc.","metadata":{}},{"cell_type":"markdown","source":"# Acknowledgements","metadata":{}},{"cell_type":"markdown","source":"I would like to thank Samira Kumar for the visualization tool to make animations of tracking data (https://github.com/samirak93/Game-Animation).","metadata":{}}]}