{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Punt Commons\nI wanted to take the challenge of understanding the punts more. To be honest, I did not know anything about the american football before this challenge. I needed to read and watch a lot first to understand what the people are doing over there. As you will be reading in the next parts, I probably still miss some knowledge in this field. I really welcome if you would help me with my not so correct interpretations.\n\nIn this notebook, I created a model with an weighted F1 score of 0.87 with very minimal number of features, by defining a custom success. Then I tried to understand why the model behaved so and where it failed.","metadata":{}},{"cell_type":"code","source":"!pip install swifter pandarallel xgboost shap ","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2022-01-05T08:48:38.848073Z","iopub.execute_input":"2022-01-05T08:48:38.848449Z","iopub.status.idle":"2022-01-05T08:48:47.739940Z","shell.execute_reply.started":"2022-01-05T08:48:38.848408Z","shell.execute_reply":"2022-01-05T08:48:47.738835Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\n\nimport seaborn as sns \nimport matplotlib.animation as anim\nimport matplotlib.pyplot as plt\nimport plotly.express as px\nfrom plotly.subplots import make_subplots\nimport plotly.graph_objs as go\nimport ipywidgets as wg\nfrom IPython.core.display import HTML\n\nfrom pandarallel import pandarallel\nimport swifter\n\nfrom sklearn.pipeline import Pipeline\nfrom sklearn import preprocessing\nfrom sklearn.compose import ColumnTransformer\nfrom sklearn.model_selection import train_test_split, TimeSeriesSplit, StratifiedKFold, cross_val_score, GridSearchCV\nfrom sklearn.metrics import classification_report, accuracy_score, f1_score\nfrom sklearn.ensemble import RandomForestClassifier, RandomForestClassifier\nfrom sklearn.multiclass import OneVsRestClassifier\n\nfrom xgboost.sklearn import XGBClassifier\nfrom xgboost import plot_importance\n\nimport shap","metadata":{"execution":{"iopub.status.busy":"2022-01-05T08:48:55.110634Z","iopub.execute_input":"2022-01-05T08:48:55.110976Z","iopub.status.idle":"2022-01-05T08:48:57.205010Z","shell.execute_reply.started":"2022-01-05T08:48:55.110938Z","shell.execute_reply":"2022-01-05T08:48:57.204130Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.set_option('display.max_columns', None)\npd.set_option('display.min_rows', 100)\npd.set_option('display.max_rows', 200)\nplt.style.use('ggplot')\npandarallel.initialize(progress_bar=False)","metadata":{"execution":{"iopub.status.busy":"2022-01-05T08:49:05.774241Z","iopub.execute_input":"2022-01-05T08:49:05.774527Z","iopub.status.idle":"2022-01-05T08:49:05.781965Z","shell.execute_reply.started":"2022-01-05T08:49:05.774496Z","shell.execute_reply":"2022-01-05T08:49:05.780984Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"games = pd.read_csv('../input/nfl-big-data-bowl-2022/games.csv')\nplayers = pd.read_csv('../input/nfl-big-data-bowl-2022/players.csv')\nplays = pd.read_csv('../input/nfl-big-data-bowl-2022/plays.csv')\n\n## Tracking data. Dtype tricks to save some memory\ntracking = pd.DataFrame()\nfor year in range(2018,2021):\n    temp_track = pd.read_csv('../input/nfl-big-data-bowl-2022/tracking'+str(year) +'.csv')\n    tracking = tracking.append(temp_track)\n    del temp_track\n#tracking = pd.read_csv('../input/nfl-big-data-bowl-2022/tracking2018.csv')\ntracking.fillna({\"nflId\":-1, \"jerseyNumber\":-1},inplace=True)\ntracking = tracking.astype({'nflId':np.int32, 'jerseyNumber':np.int32})\nfloat_cols = tracking.select_dtypes(include=[\"float64\"]).columns.to_list()\ntracking[float_cols] = tracking[float_cols].astype(np.float16)\n\npff = pd.read_csv(\"../input/nfl-big-data-bowl-2022/PFFScoutingData.csv\")\nplays[\"gameClockMinute\"]= plays.apply(lambda row: int(row[\"gameClock\"].split(\":\")[0]), axis=1)\n\nplays = plays.merge(games, on=[\"gameId\"],how=\"left\",validate=\"m:1\")","metadata":{"execution":{"iopub.status.busy":"2022-01-05T08:49:05.783697Z","iopub.execute_input":"2022-01-05T08:49:05.783910Z","iopub.status.idle":"2022-01-05T08:51:51.123751Z","shell.execute_reply.started":"2022-01-05T08:49:05.783886Z","shell.execute_reply":"2022-01-05T08:51:51.122786Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Standardize the tracking positions as if all games are played to the direction of left.","metadata":{}},{"cell_type":"code","source":"tracking.loc[tracking[\"playDirection\"] == \"right\", \"x\"] = 120-tracking.loc[tracking[\"playDirection\"] == \"right\", \"x\"]\ntracking.loc[tracking[\"playDirection\"] == \"right\", \"y\"] = 160.0/3-tracking.loc[tracking[\"playDirection\"] == \"right\", \"y\"]","metadata":{"execution":{"iopub.status.busy":"2022-01-05T08:51:51.125224Z","iopub.execute_input":"2022-01-05T08:51:51.125479Z","iopub.status.idle":"2022-01-05T08:52:14.264500Z","shell.execute_reply.started":"2022-01-05T08:51:51.125449Z","shell.execute_reply":"2022-01-05T08:52:14.263549Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = px.pie(plays.query(\"specialTeamsPlayType=='Punt'\"), names=\"specialTeamsResult\")\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-05T08:52:14.266016Z","iopub.execute_input":"2022-01-05T08:52:14.266976Z","iopub.status.idle":"2022-01-05T08:52:15.502375Z","shell.execute_reply.started":"2022-01-05T08:52:14.266884Z","shell.execute_reply":"2022-01-05T08:52:15.501402Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"px.pie(plays.query(\"specialTeamsPlayType =='Punt'\"),names=\"specialTeamsResult\")","metadata":{"execution":{"iopub.status.busy":"2022-01-05T08:52:15.504649Z","iopub.execute_input":"2022-01-05T08:52:15.504896Z","iopub.status.idle":"2022-01-05T08:52:15.594561Z","shell.execute_reply.started":"2022-01-05T08:52:15.504866Z","shell.execute_reply":"2022-01-05T08:52:15.593770Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"end_plays = plays.merge(\n    plays.groupby(\"gameId\").playId.max().reset_index(), \n    on=[\"gameId\", \"playId\"], \n    how = \"right\")[[\"gameId\",\"playId\",\"preSnapHomeScore\",\"preSnapVisitorScore\",\"homeTeamAbbr\",\"visitorTeamAbbr\"]]","metadata":{"execution":{"iopub.status.busy":"2022-01-05T08:52:15.595870Z","iopub.execute_input":"2022-01-05T08:52:15.596166Z","iopub.status.idle":"2022-01-05T08:52:15.626289Z","shell.execute_reply.started":"2022-01-05T08:52:15.596128Z","shell.execute_reply":"2022-01-05T08:52:15.625591Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Features for Each Punt\nThe team statistics stays unchanged during a game, but the following ones change for each punt. Namely:\n+ [x] Mean of Y and count for Vises on the left and right side\n+ [x] Mean of Y and count for Gunners on the left and right side\n+ [x] X and Y of football\n+ [x] X and Y position of the punter and the returner\n+ [x] Quarter and minute (Available on plays)\n+ [x] Punt Landed\n+ [x] Punt Received","metadata":{}},{"cell_type":"code","source":"punt_plays = plays.query(\"specialTeamsPlayType =='Punt'\").copy()\npunt_plays","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2022-01-05T08:52:15.627273Z","iopub.execute_input":"2022-01-05T08:52:15.627948Z","iopub.status.idle":"2022-01-05T08:52:15.779057Z","shell.execute_reply.started":"2022-01-05T08:52:15.627910Z","shell.execute_reply":"2022-01-05T08:52:15.778049Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"punt_plays[\"touchdown\"] = punt_plays.playDescription.apply(lambda descr: False if \"TOUCHDOWN NULLIFIED\" in descr else \"TOUCHDOWN\" in descr )","metadata":{"execution":{"iopub.status.busy":"2022-01-05T08:52:15.780709Z","iopub.execute_input":"2022-01-05T08:52:15.781072Z","iopub.status.idle":"2022-01-05T08:52:15.790923Z","shell.execute_reply.started":"2022-01-05T08:52:15.781025Z","shell.execute_reply":"2022-01-05T08:52:15.789859Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Create Labels\nFirst we need to get the starting point for the possession team. But yardlineNumber is not always counted from the side of the posssesion team.","metadata":{}},{"cell_type":"code","source":"(punt_plays.possessionTeam == punt_plays.yardlineSide).value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-01-05T08:52:15.792377Z","iopub.execute_input":"2022-01-05T08:52:15.792705Z","iopub.status.idle":"2022-01-05T08:52:15.809725Z","shell.execute_reply.started":"2022-01-05T08:52:15.792659Z","shell.execute_reply":"2022-01-05T08:52:15.808620Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So new column is created to indicated yardlineNumber from the side of possesion team.\nBut when the side is not the possession team, the count should be done from the other side.\n\nAlso to indicate the yardlineNumber of the yardsToGo from the punter side, poss_yardlineNumber_toGo column is created","metadata":{}},{"cell_type":"code","source":"punt_plays[\"poss_yardlineNumber\"] = punt_plays.yardlineNumber\npunt_plays.loc[punt_plays.possessionTeam != punt_plays.yardlineSide, [\"poss_yardlineNumber\"]] = 100 - punt_plays.loc[punt_plays.possessionTeam != punt_plays.yardlineSide, [\"poss_yardlineNumber\"]]\npunt_plays[\"poss_yardlineNumber_toGo\"] = punt_plays.poss_yardlineNumber + punt_plays.yardsToGo","metadata":{"execution":{"iopub.status.busy":"2022-01-05T08:52:15.810946Z","iopub.execute_input":"2022-01-05T08:52:15.811175Z","iopub.status.idle":"2022-01-05T08:52:15.829736Z","shell.execute_reply.started":"2022-01-05T08:52:15.811148Z","shell.execute_reply":"2022-01-05T08:52:15.829002Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now let us calculate the line number from the side of the receiving team, when the play is over.","metadata":{}},{"cell_type":"code","source":"punt_plays[\"left_to_end\"] = 100-(punt_plays.poss_yardlineNumber + punt_plays.playResult)","metadata":{"execution":{"iopub.status.busy":"2022-01-05T08:52:15.830856Z","iopub.execute_input":"2022-01-05T08:52:15.831271Z","iopub.status.idle":"2022-01-05T08:52:15.837891Z","shell.execute_reply.started":"2022-01-05T08:52:15.831235Z","shell.execute_reply":"2022-01-05T08:52:15.837153Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Below, we see that:\n- Almost all Touchbaks are resulted in 20 yardline, which makes sense after the ball crosses the goal line.\n- On average Returns are resulted between 30 and 40 but we dont know it was due to punt returners' abilities or the punters aimed those area (or close to that area).\n- The fair catch calls decrease a lot within the 10 yards of the receiving team.\n- The ball is downed more in the same yards. This is actually the plot of punt returners' unfullfilled hopes.\n- Muffed is distributed a large area on the returning team side, with counts of no partical difference. Considering that this is a momentary slip of the ball from the hand, it makes sense to see a white noise like distribution.\n\nWhen it comes to labeling:\n- Within 0-20 yardlines, it will be a **good progress**\n- More than 20 yardlines will be a **bad progress**.\n- When the result is downed, muffed(if the yardsToGo is exceeded) or touchdown (with left_to_end is equal to 0) it is **good result**.\n- When the left_to_end is 100 and it is touchdown, it is **bad result**.","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize = (15,8))\nsns.histplot(\n    x = \"left_to_end\" \n    ,hue = \"specialTeamsResult\"\n    ,multiple = \"stack\"\n    ,data = punt_plays\n).set(title=\"Special Teams Result Distribution\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-05T08:52:15.839057Z","iopub.execute_input":"2022-01-05T08:52:15.839665Z","iopub.status.idle":"2022-01-05T08:52:17.158997Z","shell.execute_reply.started":"2022-01-05T08:52:15.839617Z","shell.execute_reply":"2022-01-05T08:52:17.158102Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Just to check whether the `touchdown` and `left_to_end` information is correct, we check the plot below.\nAll looks as expected, whenever we have a touchdown, the `left_to_end` is either 100","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize = (15,8))\nsns.histplot(\n    x = \"left_to_end\" \n    ,hue = \"specialTeamsResult\"\n    ,multiple = \"stack\"\n    ,bins=100\n    ,data = punt_plays.query(\"touchdown==True\")\n).set(title=\"Special Teams Result Distribution\")\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2022-01-05T08:52:17.160427Z","iopub.execute_input":"2022-01-05T08:52:17.160942Z","iopub.status.idle":"2022-01-05T08:52:18.429366Z","shell.execute_reply.started":"2022-01-05T08:52:17.160887Z","shell.execute_reply":"2022-01-05T08:52:18.428693Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def label_it(play_row):\n    if (play_row.left_to_end == 100 and play_row.touchdown) or 100-play_row.left_to_end < play_row.poss_yardlineNumber:\n        return \"bad result\"\n    \n    if (play_row.left_to_end == 0 and play_row.touchdown) or (play_row.specialTeamsResult in [\"Muffed\", \"Downed\"] and 100-play_row.left_to_end >=play_row.poss_yardlineNumber_toGo):\n        return \"good result\"\n    \n    if  play_row.left_to_end < 20:\n        return \"good progress\"\n    else:\n        return \"bad progress\"","metadata":{"execution":{"iopub.status.busy":"2022-01-05T08:52:18.432519Z","iopub.execute_input":"2022-01-05T08:52:18.433414Z","iopub.status.idle":"2022-01-05T08:52:18.439603Z","shell.execute_reply.started":"2022-01-05T08:52:18.433372Z","shell.execute_reply":"2022-01-05T08:52:18.438773Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"punt_plays[\"punt_label\"] = punt_plays.parallel_apply(label_it, axis=1)","metadata":{"execution":{"iopub.status.busy":"2022-01-05T08:52:18.440598Z","iopub.execute_input":"2022-01-05T08:52:18.440797Z","iopub.status.idle":"2022-01-05T08:52:18.933060Z","shell.execute_reply.started":"2022-01-05T08:52:18.440772Z","shell.execute_reply":"2022-01-05T08:52:18.931590Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize = (15,8))\nsns.histplot(\n    x = \"left_to_end\" \n    ,hue = \"punt_label\"\n    ,multiple = \"stack\"\n    ,data = punt_plays\n).set(title=\"Punt Label Distribution\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-05T08:52:18.935637Z","iopub.execute_input":"2022-01-05T08:52:18.936428Z","iopub.status.idle":"2022-01-05T08:52:19.776676Z","shell.execute_reply.started":"2022-01-05T08:52:18.936379Z","shell.execute_reply":"2022-01-05T08:52:19.775639Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize = (15,8))\nsns.countplot(x=\"punt_label\", data=punt_plays)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-05T08:52:19.778052Z","iopub.execute_input":"2022-01-05T08:52:19.778283Z","iopub.status.idle":"2022-01-05T08:52:19.989621Z","shell.execute_reply.started":"2022-01-05T08:52:19.778253Z","shell.execute_reply":"2022-01-05T08:52:19.988693Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Since *good result* and *bad result* do not have many representatives, let us just blend them into *good progress* and *bad progress*, respectively.","metadata":{}},{"cell_type":"code","source":"punt_plays[\"punt_label\"] = punt_plays[[\"punt_label\"]].apply(lambda row: \"bad progress\" if(row.punt_label in [\"bad result\"])  else \"good progress\" if row.punt_label in [\"good result\"] else row.punt_label,axis=1)","metadata":{"execution":{"iopub.status.busy":"2022-01-05T08:52:19.993114Z","iopub.execute_input":"2022-01-05T08:52:19.993671Z","iopub.status.idle":"2022-01-05T08:52:20.211755Z","shell.execute_reply.started":"2022-01-05T08:52:19.993630Z","shell.execute_reply":"2022-01-05T08:52:20.211107Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Final label distribution","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize = (15,8))\nsns.countplot(x=\"punt_label\", data=punt_plays)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-05T08:52:20.213067Z","iopub.execute_input":"2022-01-05T08:52:20.214045Z","iopub.status.idle":"2022-01-05T08:52:20.406873Z","shell.execute_reply.started":"2022-01-05T08:52:20.213994Z","shell.execute_reply":"2022-01-05T08:52:20.406211Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Extract punt statistics from each row of punt plays","metadata":{}},{"cell_type":"code","source":"def punt_stats(play_row):\n    attacker,returner,returnerTeam = (\"home\",\"away\", play_row.visitorTeamAbbr) if play_row.possessionTeam == play_row.homeTeamAbbr else (\"away\", \"home\",play_row.homeTeamAbbr)\n    pff_row = pff.query(\n        \"gameId=={0} and playId=={1}\".format(\n            play_row.gameId,\n            play_row.playId\n        )\n    ).iloc[0]\n    \n    the_play =  tracking.query(\n        \"gameId=={0} and playId=={1}\".format(\n            play_row.gameId,\n            play_row.playId\n        )\n    )\n    \n    play_first_frame = the_play.query(\"frameId==1\")\n    \n    football = play_first_frame.query(\"team=='football'\").iloc[0]\n    \n    # Gunners Features\n    if not pd.isna(pff_row.gunners):\n        gunners_jersey = [int(full_jersey.strip().split(\" \")[1]) for full_jersey in pff_row.gunners.split(\";\")]\n        gunners = play_first_frame[play_first_frame.jerseyNumber.isin(gunners_jersey)].query(\"team=='{0}'\".format(attacker))\n        left_gunners = gunners.query(\"y <= {0}\".format(football.y))\n        right_gunners = gunners.query(\"y > {0}\".format(football.y))\n        \n        if left_gunners.shape[0]>0:\n            left_gunners_y = left_gunners.y.mean()\n            num_left_gunners = left_gunners.shape[0]\n        else:\n            left_gunners_y=-25\n            num_left_gunners = 0\n        \n        if right_gunners.shape[0]>0:\n            right_gunners_y = right_gunners.y.mean()\n            num_right_gunners = right_gunners.shape[0]\n        else:\n            right_gunners_y=-25\n            num_right_gunners = 0\n        \n    else:\n        left_gunners_y=-25\n        num_left_gunners = 0\n        right_gunners_y=-25\n        num_right_gunners = 0 \n    \n    # Vises Features\n    if not pd.isna(pff_row.vises):\n        vises_jersey = [int(full_jersey.strip().split(\" \")[1]) for full_jersey in pff_row.vises.split(\";\")]\n        vises = play_first_frame[play_first_frame.jerseyNumber.isin(vises_jersey)].query(\"team=='{0}'\".format(returner))\n        left_vises = vises.query(\"y <= {0}\".format(football.y))\n        right_vises = vises.query(\"y > {0}\".format(football.y))\n        \n        if left_vises.shape[0]>0:\n            left_vises_y = left_vises.y.mean()\n            num_left_vises = left_vises.shape[0]\n        else:\n            left_vises_y=-25\n            num_left_vises = 0\n        \n        if right_vises.shape[0]>0:\n            right_vises_y = right_vises.y.mean()\n            num_right_vises = right_vises.shape[0]\n        else:\n            right_vises_y=-25\n            num_right_vises = 0\n        \n    else:\n        left_vises_y=-25\n        num_left_vises = 0\n        right_vises_y=-25\n        num_right_vises = 0\n        \n    \n    # Punter\n    #punter = play_first_frame[play_first_frame.y == play_first_frame.y.max()].iloc[0]\n    \n    # Punt returner\n    #punt_returner = play_first_frame[play_first_frame.y == play_first_frame.y.min()].iloc[0]\n    \n    punt_received = 1 if the_play.query(\"event=='punt_received' or event=='fair_catch'\").shape[0]>0 else 0\n    \n    punt_landed = 1 if the_play.query(\"event=='punt_land'\").shape[0]>0 else 0\n    \n    \n    # Get score difference : Punting Team - Returning Team\n    score_diff = play_row.preSnapHomeScore - play_row.preSnapVisitorScore\n    if attacker != \"home\":\n        score_diff *= -1\n    \n    return [\n        play_row.gameId \n        ,play_row.playId\n        #,punter.x\n        #,punter.y\n        ,left_gunners_y\n        ,num_left_gunners\n        ,right_gunners_y\n        ,num_right_gunners\n        ,left_vises_y\n        ,num_left_vises\n        ,right_vises_y\n        ,num_right_vises\n        #,punt_returner.nflId\n        #,punt_returner.x\n        #,punt_returner.y\n        ,punt_received\n        ,punt_landed\n        ,football.x\n        ,football.y\n        ,score_diff\n        ,returnerTeam\n    ]","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2022-01-05T08:52:20.408038Z","iopub.execute_input":"2022-01-05T08:52:20.408379Z","iopub.status.idle":"2022-01-05T08:52:20.428686Z","shell.execute_reply.started":"2022-01-05T08:52:20.408349Z","shell.execute_reply":"2022-01-05T08:52:20.427829Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"punt_feats = punt_plays.parallel_apply(punt_stats,axis=1,result_type ='expand')\npunt_feats.columns=[\n    \"gameId\"\n    ,\"playId\"\n    #,\"punter_x\"\n    #,\"punter_y\"\n    ,\"left_gunners_y\"\n    ,\"num_left_gunners\"\n    ,\"right_gunners_y\"\n    ,\"num_right_gunners\"\n    ,\"left_vises_y\"\n    ,\"num_left_vises\"\n    ,\"right_vises_y\"\n    ,\"num_right_vises\"\n    #,\"punt_returner_x\"\n    #,\"punt_returner_y\"\n    ,\"punt_received\"\n    ,\"punt_landed\"\n    ,\"football_x\"\n    ,\"football_y\"\n    ,\"score_diff\"\n    ,\"returnerTeam\"\n]","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2022-01-05T08:52:20.429805Z","iopub.execute_input":"2022-01-05T08:52:20.430519Z","iopub.status.idle":"2022-01-05T09:03:48.157643Z","shell.execute_reply.started":"2022-01-05T08:52:20.430453Z","shell.execute_reply":"2022-01-05T09:03:48.156693Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"punt_feats","metadata":{"execution":{"iopub.status.busy":"2022-01-05T09:03:48.159780Z","iopub.execute_input":"2022-01-05T09:03:48.160703Z","iopub.status.idle":"2022-01-05T09:03:48.236580Z","shell.execute_reply.started":"2022-01-05T09:03:48.160647Z","shell.execute_reply":"2022-01-05T09:03:48.235769Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"punt_data = punt_plays.reset_index()\\\n        .merge(punt_feats, how=\"inner\", on=[\"gameId\",\"playId\"])\\\n        .merge(pff[[\"gameId\", \"playId\", \"hangTime\"]])[\n        [\n            \"index\"\n            #,\"possessionTeam\"\n            #,\"returnerTeam\"\n            #,\"home_num_wins\" \n            #,\"home_num_draws\"\n            #,\"visitor_num_wins\"\n            #,\"visitor_num_draws\"\n            ,\"hangTime\"\n            ,\"left_gunners_y\"\n            ,\"num_left_gunners\"\n            ,\"right_gunners_y\"\n            ,\"num_right_gunners\"\n            ,\"left_vises_y\"\n            ,\"right_vises_y\"\n            ,\"punt_received\"\n            ,\"punt_landed\"\n            #,\"football_x\"\n            ,\"football_y\"\n            ,\"score_diff\"\n            ,\"gameClockMinute\"\n            ,\"kickLength\"\n            ,\"yardsToGo\"\n            ,\"poss_yardlineNumber\"\n            ,\"punt_label\"\n        ]\n    ]\npunt_data.set_index(\"index\", inplace=True)\npunt_data[\"hangTime\"] = punt_data.hangTime.fillna(0)\npunt_data[\"kickLength\"] = punt_data.kickLength.fillna(0.0000001)\npunt_data[\"hang_per_length\"] = punt_data.hangTime / punt_data.kickLength\npunt_data[\"kick_end\"] = punt_data.kickLength + punt_data.poss_yardlineNumber\npunt_data.dropna()","metadata":{"execution":{"iopub.status.busy":"2022-01-05T09:03:48.238108Z","iopub.execute_input":"2022-01-05T09:03:48.238850Z","iopub.status.idle":"2022-01-05T09:03:48.390747Z","shell.execute_reply.started":"2022-01-05T09:03:48.238807Z","shell.execute_reply":"2022-01-05T09:03:48.389884Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To save some time in the next runs, save the `punt_data` to a file.","metadata":{}},{"cell_type":"code","source":"punt_data.to_csv(\"punt_data.csv\")\n#punt_data = pd.read_csv(\"punt_data.csv\", index_col=0)","metadata":{"execution":{"iopub.status.busy":"2022-01-05T09:03:48.391945Z","iopub.execute_input":"2022-01-05T09:03:48.392177Z","iopub.status.idle":"2022-01-05T09:03:48.497225Z","shell.execute_reply.started":"2022-01-05T09:03:48.392148Z","shell.execute_reply":"2022-01-05T09:03:48.496293Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Preprocessing\nBecause we will be using xgboost and random forest as the models, there is no added value of monotonic scaling. The only preprocessing will be for the label encoding.","metadata":{}},{"cell_type":"code","source":"label_pipeline = Pipeline(steps=[\n  (\"label_encoder\", preprocessing.OrdinalEncoder())\n])\n#features_pipeline = Pipeline(steps=[\n#    (\"feature_scaler\", preprocessing.MinMaxScaler())\n#])","metadata":{"execution":{"iopub.status.busy":"2022-01-05T09:05:30.684885Z","iopub.execute_input":"2022-01-05T09:05:30.685222Z","iopub.status.idle":"2022-01-05T09:05:30.691224Z","shell.execute_reply.started":"2022-01-05T09:05:30.685176Z","shell.execute_reply":"2022-01-05T09:05:30.690107Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Divide the data to train and test sets, then run the pipelines","metadata":{}},{"cell_type":"code","source":"X = punt_data.drop([\"punt_label\"], axis=1)\ny = punt_data[[\"punt_label\"]]\nX_train, _, y_train_raw, _ = train_test_split(X,y, shuffle=False, test_size=0.2)\n\ny_train = label_pipeline.fit_transform(y_train_raw)","metadata":{"execution":{"iopub.status.busy":"2022-01-05T09:05:30.692752Z","iopub.execute_input":"2022-01-05T09:05:30.693007Z","iopub.status.idle":"2022-01-05T09:05:30.717882Z","shell.execute_reply.started":"2022-01-05T09:05:30.692977Z","shell.execute_reply":"2022-01-05T09:05:30.717066Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Short EDA Before Training","metadata":{}},{"cell_type":"code","source":"train = X_train.merge(pd.get_dummies(y_train_raw[\"punt_label\"], prefix=\"label\"), left_index=True, right_index=True)","metadata":{"execution":{"iopub.status.busy":"2022-01-05T09:05:30.719344Z","iopub.execute_input":"2022-01-05T09:05:30.719705Z","iopub.status.idle":"2022-01-05T09:05:30.730112Z","shell.execute_reply.started":"2022-01-05T09:05:30.719671Z","shell.execute_reply":"2022-01-05T09:05:30.729382Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.info()","metadata":{"execution":{"iopub.status.busy":"2022-01-05T09:05:30.731248Z","iopub.execute_input":"2022-01-05T09:05:30.731667Z","iopub.status.idle":"2022-01-05T09:05:30.757418Z","shell.execute_reply.started":"2022-01-05T09:05:30.731633Z","shell.execute_reply":"2022-01-05T09:05:30.756691Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.describe()","metadata":{"execution":{"iopub.status.busy":"2022-01-05T09:05:30.759448Z","iopub.execute_input":"2022-01-05T09:05:30.759801Z","iopub.status.idle":"2022-01-05T09:05:30.832830Z","shell.execute_reply.started":"2022-01-05T09:05:30.759770Z","shell.execute_reply":"2022-01-05T09:05:30.832134Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize = (15,8))\nsns.heatmap(train.corr())\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-05T09:05:30.834069Z","iopub.execute_input":"2022-01-05T09:05:30.834436Z","iopub.status.idle":"2022-01-05T09:05:31.376736Z","shell.execute_reply.started":"2022-01-05T09:05:30.834404Z","shell.execute_reply":"2022-01-05T09:05:31.375812Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Models","metadata":{}},{"cell_type":"code","source":"cv_split  = StratifiedKFold(shuffle=True, random_state=10)\nresult_models=dict()\n\nmodels = {    \n    \"random_forest\":[\n        RandomForestClassifier(n_jobs=4),\n        {\n            \"max_depth\":range(3,15,1)\n            ,\"min_samples_split\":range(2,10,2)\n        }\n    ],\n    \"xgboost\":[\n        XGBClassifier(eval_metric=\"mlogloss\", use_label_encoder=False, objective=\"binary:logistic\"), {\n            \"max_depth\":range(3,17,2)\n            ,\"min_child_weight\":range(1,6,1)\n            ,\"eta\" : np.linspace(0.1,0.2,2)\n    }],\n    \n}\n\nfor model in models:\n    gridcv = GridSearchCV(\n        estimator=models[model][0], \n        scoring=\"f1_weighted\", \n        param_grid=models[model][1], \n        cv=cv_split,\n        refit=True\n    )\n    \n    %time gridcv.fit(X_train, y_train[:,0])\n    result_models[model] = gridcv.best_estimator_\n    print(\"Model:\",model)\n    print(\"Best Params:\")\n    print(gridcv.best_params_)\n    print(\"Best Score:\")\n    print(gridcv.best_score_)\n    print(\"###########################################################################\")","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2022-01-05T09:05:31.378450Z","iopub.execute_input":"2022-01-05T09:05:31.378928Z","iopub.status.idle":"2022-01-05T09:12:23.262254Z","shell.execute_reply.started":"2022-01-05T09:05:31.378879Z","shell.execute_reply":"2022-01-05T09:12:23.261015Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"result_models[\"xgboost\"].get_booster().feature_names = X.columns.to_list()\nplot_importance(result_models[\"xgboost\"])","metadata":{"execution":{"iopub.status.busy":"2022-01-05T09:12:23.264580Z","iopub.execute_input":"2022-01-05T09:12:23.264885Z","iopub.status.idle":"2022-01-05T09:12:23.647166Z","shell.execute_reply.started":"2022-01-05T09:12:23.264849Z","shell.execute_reply":"2022-01-05T09:12:23.646203Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"label_pipeline[\"label_encoder\"].categories_","metadata":{"execution":{"iopub.status.busy":"2022-01-05T09:12:23.648774Z","iopub.execute_input":"2022-01-05T09:12:23.649041Z","iopub.status.idle":"2022-01-05T09:12:23.656034Z","shell.execute_reply.started":"2022-01-05T09:12:23.649006Z","shell.execute_reply":"2022-01-05T09:12:23.655134Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"SHAP for Random Forest model","metadata":{}},{"cell_type":"code","source":"rf_explainer = shap.Explainer(result_models[\"random_forest\"], output_names=label_pipeline[\"label_encoder\"].categories_)\nrf_shap_values = rf_explainer(pd.DataFrame(X_train, columns=X.columns.to_list()))\nrf_explanation = shap.Explanation(rf_shap_values[:, :, 1], data=X_train, feature_names=X.columns.to_list())\nshap.plots.beeswarm(rf_explanation,max_display=20)","metadata":{"execution":{"iopub.status.busy":"2022-01-05T09:12:23.657293Z","iopub.execute_input":"2022-01-05T09:12:23.657578Z","iopub.status.idle":"2022-01-05T09:12:34.194816Z","shell.execute_reply.started":"2022-01-05T09:12:23.657545Z","shell.execute_reply":"2022-01-05T09:12:34.193804Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"SHAP for XGBoost model","metadata":{}},{"cell_type":"code","source":"xgb_explainer = shap.Explainer(result_models[\"xgboost\"], output_names=label_pipeline[\"label_encoder\"].categories_)\nxgb_shap_values = xgb_explainer(pd.DataFrame(X_train, columns=X.columns.to_list()))\nshap.plots.beeswarm(xgb_shap_values, max_display=20)","metadata":{"execution":{"iopub.status.busy":"2022-01-05T09:12:34.196347Z","iopub.execute_input":"2022-01-05T09:12:34.196620Z","iopub.status.idle":"2022-01-05T09:12:36.274747Z","shell.execute_reply.started":"2022-01-05T09:12:34.196587Z","shell.execute_reply":"2022-01-05T09:12:36.273879Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## After a Simpler Model\nBefore doing any model explanation, I would like to try to get the model simpler hopefully without losing any significant accuracy or F1 score. For this purpose, my focus will be on random forest model since the SHAP value of xgboost model is more heavily dependend on a single feauture: **kick end**, while random forest model shows the same symptoms but to a smaller extend, so I think it is less prone to overfitting. In any case, the smallest mean absolute SHAP values of features are mostly similar with minor differences.\n\nStarting from bottom of the graph, I would like to put the features up to the kick end (excluding) into a simple feature selection process. Let's start with it for a final model!\n\nPlease note that when I ran on my local, the order was slightly different. It would be better I would use a random state for reproducability, but I don't want to touch it since it is not affecting the end result.","metadata":{}},{"cell_type":"code","source":"features_to_select = [\n    \"num_right_gunners\"\n    ,\"num_left_gunners\"\n    ,\"gameClockMinute\"\n    ,\"score_diff\"\n    ,\"yardsToGo\"\n    ,\"football_y\"\n    ,\"right_gunners_y\"\n    ,\"right_vises_y\"\n    ,\"left_vises_y\"\n    ,\"left_gunners_y\"\n    ,\"hangTime\"\n    ,\"kickLength\"\n    ,\"hang_per_length\"\n    ,\"punt_landed\"\n    ,\"punt_received\"\n    ,\"poss_yardlineNumber\"\n]\ncv_split  = StratifiedKFold(shuffle=True, random_state=10)\nmodel_results=[]\n\nmodels = {    \n    \"random_forest\":[\n        RandomForestClassifier(n_jobs=4),\n        {\n            \"max_depth\":range(3,15,1)\n            ,\"min_samples_split\":range(2,10,2)\n        }\n    ],\n    \"xgboost\":[\n        XGBClassifier(eval_metric=\"mlogloss\", use_label_encoder=False, objective=\"binary:logistic\"), {\n            \"max_depth\":range(3,17,2)\n            ,\"min_child_weight\":range(1,6,1)\n            ,\"eta\" : np.linspace(0.1,0.2,2)\n    }],\n    \n}\nfor n_drop in range(1,len(features_to_select)+1):\n    X = punt_data.drop(\n    [\n        \"punt_label\",\n        *features_to_select[:n_drop]\n        \n    ], axis=1\n    )\n    y = punt_data[[\"punt_label\"]]\n    X_train, _, y_train_raw, _ = train_test_split(X,y, shuffle=False, test_size=0.2)\n\n    y_train = label_pipeline.fit_transform(y_train_raw)\n\n    for model in models:\n\n        gridcv = GridSearchCV(\n            estimator=models[model][0], \n            scoring=\"f1_weighted\", \n            param_grid=models[model][1], \n            cv=cv_split,\n            refit=True\n        )\n\n        %time gridcv.fit(X_train, y_train[:,0])\n        train_preds = gridcv.best_estimator_.predict(X_train)\n        model_results.append(\n            (\n                model \n                ,n_drop\n                ,gridcv\n                ,gridcv.best_score_\n            )\n        )\n        print(\"Model:\",model, \"Num to Drop:\", n_drop)\n        print(\"Best Params:\")\n        print(gridcv.best_params_)\n        print(\"Best Score:\")\n        print(gridcv.best_score_)\n        print(\"###########################################################################\")","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2022-01-05T09:12:36.276007Z","iopub.execute_input":"2022-01-05T09:12:36.276230Z","iopub.status.idle":"2022-01-05T10:37:05.518809Z","shell.execute_reply.started":"2022-01-05T09:12:36.276195Z","shell.execute_reply":"2022-01-05T10:37:05.518094Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let us have a closer look on the results","metadata":{}},{"cell_type":"code","source":"model_res = pd.DataFrame(model_results, columns=[\"model\", \"features_dropped\", \"GridSearchObj\", \"cv_f1\"])\nmodel_scores = model_res.drop([\"GridSearchObj\"],axis=1)\nmodel_scores","metadata":{"execution":{"iopub.status.busy":"2022-01-05T10:37:05.520285Z","iopub.execute_input":"2022-01-05T10:37:05.521001Z","iopub.status.idle":"2022-01-05T10:37:05.538282Z","shell.execute_reply.started":"2022-01-05T10:37:05.520962Z","shell.execute_reply":"2022-01-05T10:37:05.537591Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(14,10))\nsns.lineplot(x=\"features_dropped\",y=\"cv_f1\", style=\"model\", data=model_scores)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-05T10:37:05.541018Z","iopub.execute_input":"2022-01-05T10:37:05.541374Z","iopub.status.idle":"2022-01-05T10:37:05.863731Z","shell.execute_reply.started":"2022-01-05T10:37:05.541339Z","shell.execute_reply":"2022-01-05T10:37:05.862807Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As seen in the above graph, after dropping 13 features the validation results does not even change. But one obvious point is that, with only `kick end` feature f1 is close to 0.865. Additional features 4 features give us only about 0.03 f1.\n\nRandom forest model looks like performing better in general also in 13-features-dropped model, so I will go on with this model.\n\n## Model Test Results\nLet us now check how this model performs and takes decisions in the test set!","metadata":{}},{"cell_type":"code","source":"n_drop=13\nX = punt_data.drop(\n    [\n        \"punt_label\",\n        *features_to_select[:n_drop]\n        \n    ], axis=1\n    )\ny = punt_data[[\"punt_label\"]]\n_, X_test, _, y_test_raw = train_test_split(X,y, shuffle=False, test_size=0.2)\n\ny_test = label_pipeline.transform(y_test_raw)","metadata":{"execution":{"iopub.status.busy":"2022-01-05T10:37:05.865148Z","iopub.execute_input":"2022-01-05T10:37:05.866206Z","iopub.status.idle":"2022-01-05T10:37:05.878713Z","shell.execute_reply.started":"2022-01-05T10:37:05.866156Z","shell.execute_reply":"2022-01-05T10:37:05.877763Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"final_model = model_res.query(\"features_dropped==13 and model=='random_forest'\").iloc[0][\"GridSearchObj\"].best_estimator_\ny_pred = final_model.predict(X_test)\nprint(classification_report(y_test, y_pred))","metadata":{"execution":{"iopub.status.busy":"2022-01-05T10:37:05.880173Z","iopub.execute_input":"2022-01-05T10:37:05.880783Z","iopub.status.idle":"2022-01-05T10:37:06.004958Z","shell.execute_reply.started":"2022-01-05T10:37:05.880735Z","shell.execute_reply":"2022-01-05T10:37:06.004108Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We have a similar score on the test set, awesome! Let us now check how these decisions are made with `shap`.","metadata":{}},{"cell_type":"markdown","source":"## Exploring the Model with SHAP","metadata":{}},{"cell_type":"code","source":"rf_explainer = shap.Explainer(final_model, output_names=label_pipeline[\"label_encoder\"].categories_)\nrf_shap_values = rf_explainer(X_test)\nrf_explanation = shap.Explanation(rf_shap_values[:, :, 1], data=X_test, feature_names=X.columns.to_list())\nshap.plots.beeswarm(rf_explanation)","metadata":{"execution":{"iopub.status.busy":"2022-01-05T10:37:06.006435Z","iopub.execute_input":"2022-01-05T10:37:06.006675Z","iopub.status.idle":"2022-01-05T10:37:06.884381Z","shell.execute_reply.started":"2022-01-05T10:37:06.006645Z","shell.execute_reply":"2022-01-05T10:37:06.883355Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Kick End\nIt is clear that `kick_end` is the most important feature of the model. Just to remind, `kick_end` is the yardline number where the ball drops (either received or landed or out of bounds) where the yardlines are counted starting from punting team end zone. In general higher is better.\n\nAs seen in the graph, higher the `kick_end` it is more likely for a good result for the punting team, up to a point. Then, the effect starts to work in the reverse direction. Let us now check the distribution in a better graph.","metadata":{}},{"cell_type":"code","source":"#fig = plt.figure(figsize=(12,10))\nshap.plots.scatter(rf_explanation[:,\"kick_end\"], show=False)\nfig = plt.gcf() \nfig.set_size_inches(12, 8)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-05T10:37:06.885794Z","iopub.execute_input":"2022-01-05T10:37:06.886037Z","iopub.status.idle":"2022-01-05T10:37:07.226992Z","shell.execute_reply.started":"2022-01-05T10:37:06.886006Z","shell.execute_reply":"2022-01-05T10:37:07.226163Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So the model is very realistic, if the punter does not punt the ball to within the 20th yardline of the receiving team, then this works against the punting team. Given the definition of *good progress* this just makes sense.\n\nBetween 80 and 100 the SHAP values increase with a constant speed. But very close to 100 and in 100, the SHAP values decrease, because the possibility of touchback is increasing and the punt returner has more room to advance the ball back.\n\n###  End of Punt Combined with the Start\nLet us check the effect of `kick_end` together with `poss_yardlineNumber`.","metadata":{}},{"cell_type":"code","source":"shap.plots.scatter(rf_explanation[:,\"kick_end\"], color=rf_explanation[:,\"poss_yardlineNumber\"], show=False, cmap=plt.get_cmap(\"cool\"))\nfig = plt.gcf() \nfig.set_size_inches(12, 8)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-05T10:37:07.229181Z","iopub.execute_input":"2022-01-05T10:37:07.229457Z","iopub.status.idle":"2022-01-05T10:37:07.672336Z","shell.execute_reply.started":"2022-01-05T10:37:07.229410Z","shell.execute_reply":"2022-01-05T10:37:07.671382Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"On the right part of the graph, we can see that SHAP value is not really dependent on the start of the punt. \n\nThe story is different on the part where `kick_end < 80`. The lower the poss_yardlineNumber the SHAP value is less negative. I believe this is caused by the hang time, since it is higher when the punt started closer to the punting side and giving more chance to tackle the punt returner.The SHAP starts be very negative with higher `poss_yardlineNumber`, which could be regarded as failure since the start point is close to kick end. What I would expect is that the kick would be more precise, closer to the end zone when punted from a closer point. Plus, the hang time is lower when the punt is kicked from a point closer to the receiving team, opening a shorter time window for gunners, to reach to the punt returner before the ball comes back to the ground. According to the model, it looks like the punting team is better punt the ball to the further point possible of course under 100th yardline :) This becomes even more critical when punting from a further point.","metadata":{}},{"cell_type":"markdown","source":"### Punt Landed and Received","metadata":{}},{"cell_type":"code","source":"fig_main, axes = plt.subplots(ncols=2)\nshap.plots.scatter(rf_explanation[:,\"kick_end\"], color=rf_explanation[:,\"punt_landed\"], show=False, ax = axes[0], cmap=plt.get_cmap(\"cool\"))\nshap.plots.scatter(rf_explanation[:,\"kick_end\"], color=rf_explanation[:,\"punt_received\"], show=False, ax = axes[1], cmap=plt.get_cmap(\"cool\"))\nfig_main.set_size_inches(16, 6)\nfig_main.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-05T10:37:07.673645Z","iopub.execute_input":"2022-01-05T10:37:07.673901Z","iopub.status.idle":"2022-01-05T10:37:08.517142Z","shell.execute_reply.started":"2022-01-05T10:37:07.673868Z","shell.execute_reply":"2022-01-05T10:37:08.516370Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For the part between 80 and 100, the top line where the punt is neither received nor landed would mean it is out of bounds. So it is a direct gain without any other possibility, that is why we see this small line with higher SHAP values. For the other 2 parts where it is either received or landed, closer to `kick_end: 80` punt received has a lower SHAP than punt landed. As the `kick_end` increases, SHAP value of the `punt_received=True` increases more. So the model prefers that the punt returner let the ball bounce within last 10 yardlines, to give a slightly lower SHAP to the *good result*.\n\nBetween 0 and 55 for the `kick_end`, the line below is mostly neither received nor landed. I believe the model treats them as unsuccessful exections since the SHAP is very low. I guess, they could be short punts, blocked punts or even fake punts.\n\nFor the cluster between 40 and 80: It is clear that punt landed is preferable to punt received for the punting team, as the negative SHAP is closer to 0. The below cluster, where the punt is received, could be explained by the `poss_yardlineNumber`: if you give more time to gunners your chances increase. But also by the `kick_end` itself since closer to the 80 yardline is of course better.\n\n### Focus on the Start of Punt","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(14,8))\nnorm = plt.Normalize(rf_explanation[:,\"kick_end\"].data.min(), rf_explanation[:,\"kick_end\"].data.max())\nsm = plt.cm.ScalarMappable(cmap=plt.get_cmap(\"tab20\"), norm=norm)\nsm.set_array([])\nax = sns.scatterplot(x=rf_explanation[:,\"poss_yardlineNumber\"].data, y=rf_explanation[:,\"poss_yardlineNumber\"].values, hue=rf_explanation[:,\"kick_end\"].data, palette=plt.get_cmap(\"tab20\"))\nax.get_legend().remove()\nax.figure.colorbar(sm, label=\"kick_end\")\nax.set_ylabel('SHAP Value for poss_yardlineNumber')\nax.set_xlabel('poss_yardlineNumber')\n\nplt.show()\n\n\nfig_main, axes = plt.subplots(ncols=2)\nshap.plots.scatter(rf_explanation[:,\"poss_yardlineNumber\"], color=rf_explanation[:,\"punt_landed\"], show=False, ax = axes[0], cmap=plt.get_cmap(\"cool\"))\nshap.plots.scatter(rf_explanation[:,\"poss_yardlineNumber\"], color=rf_explanation[:,\"punt_received\"], show=False, ax = axes[1], cmap=plt.get_cmap(\"cool\"))\nfig_main.set_size_inches(16, 6)\nfig_main.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-05T10:37:08.518760Z","iopub.execute_input":"2022-01-05T10:37:08.519320Z","iopub.status.idle":"2022-01-05T10:37:10.159515Z","shell.execute_reply.started":"2022-01-05T10:37:08.519248Z","shell.execute_reply":"2022-01-05T10:37:10.158825Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I am actually fascinated that you can draw a dinosaur with SHAP values of a random forest model. In this one what really grabbed my attention is that when the `kick_end` is between 80 and 90, the SHAP value from `poss_yardlineNumber` is even lower than the tail of the dinosaur (where the `kick_end` is mostly below 80). For the values between 80 and 90, we almost have a linear function, or tanh to be more precise. The closer the start point to the receiving team's endzone, the better the chances. Around 40 `poss_yardlineNumber` is a turning point from negative SHAP to a positive. Maybe this has something to do with that mostly they are received, not landed. I might be missing some field knowledge, so I would appreciate if you could help in the comments.\n\n### Another Check on the Interpretations\nI tried to interpret the model, but this is like accepting trends are all correct. Now I would like to do one more visualization to catch, where we can see the points it failed.","metadata":{}},{"cell_type":"code","source":"X_check = X_test.copy()\nX_check[\"result_correct\"] = (y_pred == y_test.squeeze())","metadata":{"execution":{"iopub.status.busy":"2022-01-05T10:37:10.160752Z","iopub.execute_input":"2022-01-05T10:37:10.161595Z","iopub.status.idle":"2022-01-05T10:37:10.166964Z","shell.execute_reply.started":"2022-01-05T10:37:10.161547Z","shell.execute_reply":"2022-01-05T10:37:10.165696Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#plt.figure(figsize=(14,8))\nfig, axes = plt.subplots(ncols=2,figsize=(16,6))\nsns.scatterplot(x=rf_explanation[:,\"kick_end\"].data, y=rf_explanation[:,\"kick_end\"].values, hue=X_check[\"result_correct\"], ax=axes[0])\naxes[0].set_ylabel('SHAP Value for kick_end')\naxes[0].set_xlabel('kick_end')\n\nshap.plots.scatter(rf_explanation[:,\"kick_end\"], color=rf_explanation[:,\"poss_yardlineNumber\"], show=False, cmap=plt.get_cmap(\"cool\"), ax=axes[1])\n\nplt.show()\n\nfig_landed, axes_l = plt.subplots(ncols=2)\nshap.plots.scatter(rf_explanation[:,\"kick_end\"], color=rf_explanation[:,\"punt_landed\"], show=False, ax = axes_l[0], cmap=plt.get_cmap(\"cool\"))\nshap.plots.scatter(rf_explanation[:,\"kick_end\"], color=rf_explanation[:,\"punt_received\"], show=False, ax = axes_l[1], cmap=plt.get_cmap(\"cool\"))\nfig_landed.set_size_inches(16, 6)\nfig_landed.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-05T10:37:10.167996Z","iopub.execute_input":"2022-01-05T10:37:10.168207Z","iopub.status.idle":"2022-01-05T10:37:11.687617Z","shell.execute_reply.started":"2022-01-05T10:37:10.168179Z","shell.execute_reply":"2022-01-05T10:37:11.686713Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So a clear pattern is in the cluster between 40-80 `kick_end` where the landed is preferred to the received mentioned. A big portion of the mispredictions are in section. Therefore this comment is actually not so correct, and to be improved with more/better data or features.\n\nAnother suspicious part is between 80 and 100, it looks like mostly the received punts are incorrect, even though it does not look bad as in the previous part. For this one, I think it is better if we see the numbers.","metadata":{}},{"cell_type":"code","source":"X_check.query(\"kick_end>80\").groupby([\"punt_received\", \"result_correct\"]).count()","metadata":{"execution":{"iopub.status.busy":"2022-01-05T10:37:11.688833Z","iopub.execute_input":"2022-01-05T10:37:11.689080Z","iopub.status.idle":"2022-01-05T10:37:11.707201Z","shell.execute_reply.started":"2022-01-05T10:37:11.689049Z","shell.execute_reply":"2022-01-05T10:37:11.706479Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"92/(92+290)*100","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2022-01-05T10:49:41.142822Z","iopub.execute_input":"2022-01-05T10:49:41.143331Z","iopub.status.idle":"2022-01-05T10:49:41.150603Z","shell.execute_reply.started":"2022-01-05T10:49:41.143262Z","shell.execute_reply":"2022-01-05T10:49:41.149761Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The suspicions are correct, when the punt is received the predictions are wrong around 24%, after the 80th yardline. However this is not changing the idea that the model prefers punt returner to bounce the ball. I believe the model is not taking the possibilty of punt returners options or capabilities. More features like the number of gunners close to the ball when it is received or player features like speed acceleration of the punt returner could help over here.\n\n## Future Work\nI believe the success could be made more sophisticated. If I would have more data, I really wanted to keep *good result* and *bad result* in the label to get more insight.\nAlso, there could be more features to help the success of the model. As mentioned there is a clear pattern in the models mispredictions. This could be eliminated with good features.\n\nThank you for reading :)\n\nIf you want to reach me, here it is https://www.linkedin.com/in/akyazi/","metadata":{}}]}