{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2021-09-28T18:02:42.292826Z","iopub.execute_input":"2021-09-28T18:02:42.294204Z","iopub.status.idle":"2021-09-28T18:02:42.306216Z","shell.execute_reply.started":"2021-09-28T18:02:42.294144Z","shell.execute_reply":"2021-09-28T18:02:42.305514Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Games data\n\nLoad the Games data to answer questions like:\n- How many games are in the data set?\n- How many games from each season?\n\nThe answers will be used to put special-team penalties in perspective (penalties per game, penalty yards per game, etc.).","metadata":{}},{"cell_type":"code","source":"game_data = pd.read_csv('/kaggle/input/nfl-big-data-bowl-2022/games.csv')\ngame_data.head()","metadata":{"execution":{"iopub.status.busy":"2021-09-28T18:02:42.307642Z","iopub.execute_input":"2021-09-28T18:02:42.308648Z","iopub.status.idle":"2021-09-28T18:02:42.333397Z","shell.execute_reply.started":"2021-09-28T18:02:42.308599Z","shell.execute_reply":"2021-09-28T18:02:42.332430Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(game_data.shape)\ngame_data.groupby('season')['gameId'].count()","metadata":{"execution":{"iopub.status.busy":"2021-09-28T18:02:42.334810Z","iopub.execute_input":"2021-09-28T18:02:42.335743Z","iopub.status.idle":"2021-09-28T18:02:42.345732Z","shell.execute_reply.started":"2021-09-28T18:02:42.335703Z","shell.execute_reply":"2021-09-28T18:02:42.344528Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Plays data\n\nLoad the Plays data set to answer questions like:\n- [How many total special-teams plays are in the data set?](#plays_q1)\n- [How many penalties were called?](#plays_q2)\n- [What proportion of special-teams plays resulted in a penalty?](#plays_q3)\n- [What are the most penalized teams in the NFL?](#plays_q4)\n- [Which types of penalties are the most common on special-teams plays?](#plays_q5)\n- [What type of penalty is most common for each play type?](#plays_q6)\n- [Which play type resulted in the most penalty yards?](#plays_q7)\n- [What is the distribution of penalty yards for each play type?](#plays_q8)","metadata":{}},{"cell_type":"code","source":"play_data = pd.read_csv('/kaggle/input/nfl-big-data-bowl-2022/plays.csv')\nplay_data.head()","metadata":{"execution":{"iopub.status.busy":"2021-09-28T18:02:42.348332Z","iopub.execute_input":"2021-09-28T18:02:42.349129Z","iopub.status.idle":"2021-09-28T18:02:42.478743Z","shell.execute_reply.started":"2021-09-28T18:02:42.349090Z","shell.execute_reply":"2021-09-28T18:02:42.477864Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id='plays_q1'></a>\n#### How many total special-teams plays are in the data set?","metadata":{}},{"cell_type":"code","source":"print(play_data.shape)","metadata":{"execution":{"iopub.status.busy":"2021-09-28T18:02:42.480058Z","iopub.execute_input":"2021-09-28T18:02:42.481109Z","iopub.status.idle":"2021-09-28T18:02:42.487096Z","shell.execute_reply.started":"2021-09-28T18:02:42.481061Z","shell.execute_reply":"2021-09-28T18:02:42.485913Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are 19,979 toal special-teams plays in the data set.","metadata":{}},{"cell_type":"code","source":"play_data.describe()","metadata":{"execution":{"iopub.status.busy":"2021-09-28T18:02:42.488605Z","iopub.execute_input":"2021-09-28T18:02:42.488892Z","iopub.status.idle":"2021-09-28T18:02:42.557974Z","shell.execute_reply.started":"2021-09-28T18:02:42.488859Z","shell.execute_reply":"2021-09-28T18:02:42.557003Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"obj_cols = ['playDescription', 'possessionTeam', 'specialTeamsPlayType', 'specialTeamsResult',\n            'penaltyCodes', 'penaltyJerseyNumbers', 'passResult']\nplay_data[obj_cols].describe()","metadata":{"execution":{"iopub.status.busy":"2021-09-28T18:02:42.559454Z","iopub.execute_input":"2021-09-28T18:02:42.559710Z","iopub.status.idle":"2021-09-28T18:02:42.620869Z","shell.execute_reply.started":"2021-09-28T18:02:42.559681Z","shell.execute_reply":"2021-09-28T18:02:42.620059Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id='plays_q2'></a>\n#### How many penalties were called?","metadata":{}},{"cell_type":"code","source":"penalty_cols = ['playDescription', 'specialTeamsPlayType', 'possessionTeam',\n                'penaltyCodes', 'penaltyYards', 'penaltyJerseyNumbers']\npenalty_df = play_data.loc[play_data['penaltyCodes'].notnull(), penalty_cols]\npenalty_df","metadata":{"execution":{"iopub.status.busy":"2021-09-28T18:02:42.622020Z","iopub.execute_input":"2021-09-28T18:02:42.622299Z","iopub.status.idle":"2021-09-28T18:02:42.644927Z","shell.execute_reply.started":"2021-09-28T18:02:42.622263Z","shell.execute_reply":"2021-09-28T18:02:42.643851Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"A total of 1,102 penalties were called on 19,979 plays.","metadata":{}},{"cell_type":"markdown","source":"<a id='plays_q3'></a>\n#### What proportion of special-teams plays resulted in a penalty?","metadata":{}},{"cell_type":"code","source":"play_data['specialTeamsPlayType'].value_counts().plot.barh(figsize=(6,3),\n                                                           title=\"Total special-teams plays by type\")","metadata":{"execution":{"iopub.status.busy":"2021-09-28T18:02:42.645967Z","iopub.execute_input":"2021-09-28T18:02:42.646207Z","iopub.status.idle":"2021-09-28T18:02:42.878226Z","shell.execute_reply.started":"2021-09-28T18:02:42.646159Z","shell.execute_reply":"2021-09-28T18:02:42.877156Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"penalty_df['specialTeamsPlayType'].value_counts().plot.barh(figsize=(6,3), title=\"Special-teams penalties by type\")","metadata":{"execution":{"iopub.status.busy":"2021-09-28T18:02:42.880957Z","iopub.execute_input":"2021-09-28T18:02:42.881199Z","iopub.status.idle":"2021-09-28T18:02:43.096483Z","shell.execute_reply.started":"2021-09-28T18:02:42.881171Z","shell.execute_reply":"2021-09-28T18:02:43.095666Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"(penalty_df['specialTeamsPlayType'].value_counts() / play_data['specialTeamsPlayType'].value_counts()).plot.barh(figsize=(6,3),\n                                                                                                                 title=\"Proportion of penalties by play type\")","metadata":{"execution":{"iopub.status.busy":"2021-09-28T18:02:43.097743Z","iopub.execute_input":"2021-09-28T18:02:43.097973Z","iopub.status.idle":"2021-09-28T18:02:43.309937Z","shell.execute_reply.started":"2021-09-28T18:02:43.097948Z","shell.execute_reply":"2021-09-28T18:02:43.309367Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Since there are 1,102 total penalties in the data set for 19,979 plays, about 5.5% of special-teams plays overall result in a penalty. Here's how it breaks down by play type:\n- Over 10% of punt plays in the past three seasons have resulted in a penalty.\n- Less than 5% of kickoffs have resulted in penalties.\n- Less than 2% of field goal and extra point attempts result in penalties.","metadata":{}},{"cell_type":"markdown","source":"<a id='plays_q4'></a>\n#### What are the most penalized teams in the NFL?\n\nTo answer this question, we can look at the `penaltyJerseyNumbers` column, since it contains the team as a 2 or 3-letter prefix. We'll first create a new column `penaltyTeam` to isolate just the information we're interested in.","metadata":{}},{"cell_type":"code","source":"penalty_df['penaltyTeam'] = penalty_df['penaltyJerseyNumbers'].str[0:3].str.strip()\npenalty_df","metadata":{"execution":{"iopub.status.busy":"2021-09-28T18:02:43.311040Z","iopub.execute_input":"2021-09-28T18:02:43.311237Z","iopub.status.idle":"2021-09-28T18:02:43.334422Z","shell.execute_reply.started":"2021-09-28T18:02:43.311213Z","shell.execute_reply":"2021-09-28T18:02:43.333479Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now to find the most penalized teams we can group by the new column and count the results.","metadata":{}},{"cell_type":"code","source":"team_penalty_df = penalty_df.groupby('penaltyTeam')['playDescription'].count().sort_values(ascending=True)\nteam_penalty_df.plot.barh(figsize=(12,10), title=\"Special-teams penalties by Team\")\nplt.xlabel('Penalties')\nplt.ylabel('Team')","metadata":{"execution":{"iopub.status.busy":"2021-09-28T18:02:43.335964Z","iopub.execute_input":"2021-09-28T18:02:43.336348Z","iopub.status.idle":"2021-09-28T18:02:43.770382Z","shell.execute_reply.started":"2021-09-28T18:02:43.336305Z","shell.execute_reply":"2021-09-28T18:02:43.769515Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Jacksonville, Pittsburgh, and Cleveland had the most penalties called on special-teams plays over the last three season. Note that the Las Vegas Raiders had the fewest, but that's due to the fact that the team moved from Oakland between seasons. (We could combine the two by changing the labels for one or both teams if we wanted to do deeper analysis.)\n\nLet's see which teams were most penalized in terms of total yards.","metadata":{}},{"cell_type":"code","source":"penalty_df['penaltyYards'] = penalty_df['penaltyYards'].abs()\nteam_penalty_yards_df = penalty_df.groupby('penaltyTeam')['penaltyYards'].sum().sort_values(ascending=True)\n\nteam_penalty_yards_df.plot.barh(figsize=(12,10), title=\"Special-teams penalty yards by Team\")\nplt.xlabel('Penalty Yards')\nplt.ylabel('Team')","metadata":{"execution":{"iopub.status.busy":"2021-09-28T18:12:25.664526Z","iopub.execute_input":"2021-09-28T18:12:25.664820Z","iopub.status.idle":"2021-09-28T18:12:26.113296Z","shell.execute_reply.started":"2021-09-28T18:12:25.664787Z","shell.execute_reply":"2021-09-28T18:12:26.112166Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"That didn't change things by a lot. Jacksonville is still at the top, but Detroit jumped up a few spots to #2. Las Vegas, as expected is still last.","metadata":{}},{"cell_type":"markdown","source":"<a id='plays_q5'></a>\n#### Which types of penalties are the most common on special-teams plays?","metadata":{}},{"cell_type":"code","source":"unique_penalties = penalty_df['penaltyCodes'].unique()\nprint(len(unique_penalties))\nunique_penalties","metadata":{"execution":{"iopub.status.busy":"2021-09-28T18:02:43.771394Z","iopub.execute_input":"2021-09-28T18:02:43.771612Z","iopub.status.idle":"2021-09-28T18:02:43.779708Z","shell.execute_reply.started":"2021-09-28T18:02:43.771587Z","shell.execute_reply":"2021-09-28T18:02:43.778822Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Since there are 71 different penalty codes in this dataset, and many plays resulted in multiple penalties, let's reduce the number of unique values by combining multiple calls into one group.","metadata":{}},{"cell_type":"code","source":"penalty_df['penaltyCodes'].str.contains(';').sum()","metadata":{"execution":{"iopub.status.busy":"2021-09-28T18:02:43.780787Z","iopub.execute_input":"2021-09-28T18:02:43.781016Z","iopub.status.idle":"2021-09-28T18:02:43.792780Z","shell.execute_reply.started":"2021-09-28T18:02:43.780987Z","shell.execute_reply":"2021-09-28T18:02:43.791882Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"penalty_codes = penalty_df['penaltyCodes'].copy()\npenalty_codes.loc[penalty_codes.str.contains(';')] = 'Multiple'\npenalty_codes.value_counts().plot.bar(figsize=(12,3))","metadata":{"execution":{"iopub.status.busy":"2021-09-28T18:02:43.793719Z","iopub.execute_input":"2021-09-28T18:02:43.793980Z","iopub.status.idle":"2021-09-28T18:02:44.280883Z","shell.execute_reply.started":"2021-09-28T18:02:43.793954Z","shell.execute_reply":"2021-09-28T18:02:44.279916Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can refer to the [NFL Rulebook](https://operations.nfl.com/the-rules/2021-nfl-rulebook/#table-of-foul-codes) to see what the most often called penalty codes mean.\n- **OH** - Offensive Holding\n- **IBW** - Illegal Block Above the Waist\n- **UNRd** - Unnecessary Roughness\n- **ILF** - Illegal Formation\n- **DOF** - Defensive Offside\n- **UNR** - Unnecessary Roughness","metadata":{}},{"cell_type":"markdown","source":"<a id='plays_q6'></a>\n#### What type of penalties are most common for each play type?","metadata":{}},{"cell_type":"code","source":"play_type_penalties_df = penalty_df[['specialTeamsPlayType', 'penaltyCodes']].copy()\nplay_type_penalties_df['count'] = 1\nplay_type_penalty_counts = play_type_penalties_df.groupby(['specialTeamsPlayType', 'penaltyCodes'])['count'].sum().reset_index()\nplay_type_penalty_counts.sort_values(['specialTeamsPlayType','count'],ascending=False).groupby('specialTeamsPlayType').head(2)","metadata":{"execution":{"iopub.status.busy":"2021-09-28T18:02:44.281960Z","iopub.execute_input":"2021-09-28T18:02:44.282169Z","iopub.status.idle":"2021-09-28T18:02:44.305330Z","shell.execute_reply.started":"2021-09-28T18:02:44.282144Z","shell.execute_reply":"2021-09-28T18:02:44.304474Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Offensive Holding is the most common penalty called on Punt and Kickoff plays, while Defensive Offside is most common on Field Goal and Extra Point attempts.","metadata":{}},{"cell_type":"markdown","source":"<a id='plays_q7'></a>\n#### Which play type resulted in the most penalty yards?","metadata":{}},{"cell_type":"code","source":"penalty_yards_df = penalty_df[['specialTeamsPlayType', 'penaltyYards']].copy()\npenalty_yards_df['penaltyYards'] = penalty_yards_df['penaltyYards'].abs()\n\nplay_type_penalty_yards = penalty_yards_df.groupby('specialTeamsPlayType')['penaltyYards'].agg(['count', 'sum', 'mean'])\nplay_type_penalty_yards","metadata":{"execution":{"iopub.status.busy":"2021-09-28T18:02:44.306756Z","iopub.execute_input":"2021-09-28T18:02:44.307011Z","iopub.status.idle":"2021-09-28T18:02:44.325877Z","shell.execute_reply.started":"2021-09-28T18:02:44.306985Z","shell.execute_reply":"2021-09-28T18:02:44.324565Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Punt plays resulted in by far the most penalty yards on special-teams plays with 5,730. Field Goals resulted in the fewest. We'll see next why Field Goal attempts resulted in so few yards penalized when we look at the distributions.","metadata":{}},{"cell_type":"markdown","source":"<a id='plays_q8'></a>\n#### What is the distribution of penalty yards for each play type?","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(12,6))\n\nsns.kdeplot(data=penalty_yards_df.loc[penalty_yards_df['specialTeamsPlayType'] == 'Extra Point', 'penaltyYards'],\n             label=\"Extra Point\", shade=True)\nsns.kdeplot(data=penalty_yards_df.loc[penalty_yards_df['specialTeamsPlayType'] == 'Field Goal', 'penaltyYards'],\n             label=\"Field Goal\", shade=True)\nsns.kdeplot(data=penalty_yards_df.loc[penalty_yards_df['specialTeamsPlayType'] == 'Kickoff', 'penaltyYards'],\n             label=\"Kickoff\", shade=True)\nsns.kdeplot(data=penalty_yards_df.loc[penalty_yards_df['specialTeamsPlayType'] == 'Punt', 'penaltyYards'],\n             label=\"Punt\", shade=True)\n\nplt.title(\"Distribution of Penalty Yards by Play Type\")\nplt.legend()","metadata":{"execution":{"iopub.status.busy":"2021-09-28T18:02:44.326751Z","iopub.execute_input":"2021-09-28T18:02:44.326987Z","iopub.status.idle":"2021-09-28T18:02:44.900479Z","shell.execute_reply.started":"2021-09-28T18:02:44.326955Z","shell.execute_reply":"2021-09-28T18:02:44.899906Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Kickoffs and Punts both had a peak at penalties of 10 yards. Extra Point attepts had two peaks at 5 and 15 yeards, while Field Goal attempts had a high number of 0-yard penalties. That's probably worth further investigation.","metadata":{}},{"cell_type":"code","source":"cols = ['playDescription', 'specialTeamsResult', 'penaltyCodes', 'penaltyYards']\nzero_yard_penalty = play_data.loc[(play_data['specialTeamsPlayType'] == 'Field Goal') & (play_data['penaltyYards'] == 0.0), cols]\nzero_yard_penalty","metadata":{"execution":{"iopub.status.busy":"2021-09-28T18:02:44.901347Z","iopub.execute_input":"2021-09-28T18:02:44.902043Z","iopub.status.idle":"2021-09-28T18:02:44.925162Z","shell.execute_reply.started":"2021-09-28T18:02:44.902013Z","shell.execute_reply":"2021-09-28T18:02:44.924348Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"These are mostly defensive penalties where the result was 'Kick Attempt Good'. The one exception was an offensive penalty on a kick that was missed. If you read the play description, you'll see that each of these penalties was declined, which fully explains the `penaltyYards = 0.0` anomaly.","metadata":{}}]}