{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Generating Helpful Punt-Specific Features\n\nOftentimes, a player's listed position in the Data Bowl 2022 dataset is not his position on a punt play. For instance, a player may _usually_ play Cornerback (CB), but on a punt play he is a Jammer. Therefore it is challenging to do jammer-specific analysis since the players are not listed under the necessary positional name.\n\n![Punt Position](https://www.dummies.com/wp-content/uploads/283934.image0.jpg)\n\nBased on the above image of the Punt formation [from this article](https://www.dummies.com/sports/football/special-teams/football-special-teams-players-on-a-punt-team/#:~:text=Wings%3A%20The%20players%20on%20both,breaking%20free%20inside%20of%20them.), we have the following punt-specific offensive positions:\n- **Punter (P)** defined by the person who punts the ball. _this one is easy to label since often punters only play punter_\n- **Wings (W)** defined by the edges of the offensive line that are slightly off the line of scrimmage\n- **Personal Protector (PP)** defined by being 3-5 yards behind the line of scrimmage and within the edges of the offensive line\n- **Ends (E)** defined by being quite a few yards away from the edges of the offensive line. This position is optional on a given punt formation. I will be calling the **Gunner (G)** because that is the name I prefer (_sorry not sorry_)\n\nWe also have a few punt-specific defensive positions, both are optional depending on the punt formation:\n- **Punter Returner (PR)** defined by being 20+ yards away from the line of scrimmage so that they may field a punt\n- **Jammer (J)** defined by lining up to block the Gunners on the edges of the football field\n\n#### This notebook will generate a player-level table `puntPlayTable` with the following columns:\n- `gameId` Game identifier, unique (numeric)\n- `playId` Play identifier, not unique across games (numeric)\n- `nflId`  Player identification number, unique across players (numeric)\n- `isOffense` Flag if the playerId is part of the offensive punt formation or not (boolean)\n    - _this important fact is obfuscated by the way DataBowl Data is structured so this feature helped me make quicker spot checks_\n- `losX` Absolute x value of the line of scrimmage (numeric)\n    - _this important fact is obfuscated by the way DataBowl Data is structured so this feature helped me make quicker spot checks_\n- `puntPosition`  Punt-specific position a player is occupying (string)\n    - Values include the positions mentioned about (`P`,`W`, `PP`, `G`, `PR`,`J`). If a player does not fall into any of the above categories, they will be given the signature `OL` for offensive line and `DL` for deffensive line \n    - `DL` does not mean that the defensive player rushed on the play, just that they were in the box at the moment the ball was snapped","metadata":{}},{"cell_type":"code","source":"## TODO: Use the punterId flag on play\n\nimport numpy as np\nimport pandas as pd \n\nimport os\n\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","metadata":{"execution":{"iopub.status.busy":"2021-11-18T21:44:06.408928Z","iopub.execute_input":"2021-11-18T21:44:06.409634Z","iopub.status.idle":"2021-11-18T21:44:06.440122Z","shell.execute_reply.started":"2021-11-18T21:44:06.409522Z","shell.execute_reply":"2021-11-18T21:44:06.439487Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# i:\n    # trackingData || player-tracking-level data \n    # playData || play-level data \n    # gameData || game-level data\n    # verbose || if we want to see processing status\n    # allFeatures || if we want all feautres, not just the ones generated in this NB\n# o:\n    # puntPlayPd || player-level data with features mentioned in Notebook markdown\ndef generatePuntFeatures(trackingData, playData, gameData, verbose=True, allFeatures=False):\n    # Filter down to punt plays\n    if verbose:\n        print(\"Filtering down to punt plays...\")\n        \n    puntPlays = playData[playData.specialTeamsPlayType == 'Punt'][['gameId','playId']] \\\n                 .merge(trackingData, on=['gameId','playId'])\n\n    # Filter down to first frame, since \n    # we can define punt positions on coordinates at first frame\n    puntPlays = puntPlays[puntPlays.frameId==1]\n    \n    if verbose:\n        print(f\"{len(puntPlays[['gameId','playId']].drop_duplicates()):,} punt plays to process...\")\n        \n    #############################\n    ## Define Helper Features  ##\n    ############################\n    \n    if verbose:\n        print(\"Building Helper Features...\")\n    \n    # Join gameData (home Abbr) to playData (possession team), make poss_team column (home/away), then\n    # Join to trackingData (team) and make isOffense column\n    # Also create the losX column\n    gameData = gameData[['gameId','homeTeamAbbr']]\n    gamePlay = gameData.merge(playData, on=['gameId']) \\\n                         .rename(columns={'absoluteYardlineNumber':'losX'})\n    gamePlay['isPossessionHome'] = gamePlay.possessionTeam == gamePlay.homeTeamAbbr\n    gamePlay = gamePlay[['gameId','playId','isPossessionHome','losX']]\n    \n    puntPlays = puntPlays.merge(gamePlay, on=['gameId','playId'])\n    puntPlays['isOffense'] = (puntPlays.team == 'home') == (puntPlays.isPossessionHome)\n    puntPlays.loc[puntPlays.team=='football', 'isOffense'] = np.nan\n    \n    #############################\n    ### Define Punt Positions ###\n    ############################\n    \n    if verbose:\n        print(\"Building Punt Position Features 1/2 (P,G,J,PR)...\")\n    \n    # Note: using the isOffense filter on all punt position definitions\n    # is intentional because it makes clearer what the assumptions \n    # are for defining each position\n    \n    # Note: The thresholds of >40 | <17 were unscientific and came\n    # from me visualizing a bunch of plays and seeing where the \n    # gunner/jammers lined up. Leave a comment if there is a better threshold!\n    \n    puntPlays['puntPosition'] = np.nan\n\n    # Punter is easy, based on position \n    puntPlays.loc[puntPlays.position== \"P\", \"puntPosition\"] = 'P'\n\n    # Gunner is any OFF players near the sidelines \n    # Defining ABSOLUTELY because some punts do not have this position\n    puntPlays.loc[((puntPlays.y>40) | (puntPlays.y<17)) & \\\n                    (puntPlays.isOffense == True),\"puntPosition\"] = 'G'\n\n    # Jammer is any DEF players near the sidelines\n    # Defining ABSOLUTELY because some punts do not have this position\n    puntPlays.loc[((puntPlays.y>40) | (puntPlays.y<17)) & \\\n                    (puntPlays.isOffense == False),\"puntPosition\"] = 'J'\n\n    # Returner is DEF player at least 20yrds from LOS\n    # Defining ABSOLUTELY because some punts do not have this position\n    puntPlays.loc[(np.abs(puntPlays.x-puntPlays.losX)>20)&(puntPlays.isOffense == False), \"puntPosition\"] = 'PR'\n    \n    if verbose:\n        print(\"Building Punt Position Features 2/2 (W,PP,OL,DL)...\")\n\n    # Wings + PP are defined by relation to other linemen, so need to find edges + back of line\n    oLineData = puntPlays[(puntPlays.isOffense==True)&(pd.isna(puntPlays.puntPosition))] \\\n                        .groupby(['gameId','playId']).agg({\"y\":[\"min\",\"max\"],\"x\":[\"min\",\"max\"]})\n\n    oLineData['olineMinY'] = oLineData.loc[:,pd.IndexSlice[\"y\",'min']]\n    oLineData['olineMaxY'] = oLineData.loc[:,pd.IndexSlice[\"y\",'max']]\n    oLineData['olineMinX'] = oLineData.loc[:,pd.IndexSlice[\"x\",'min']]\n    oLineData['olineMaxX'] = oLineData.loc[:,pd.IndexSlice[\"x\",'max']]\n    oLineData.columns = oLineData.columns.droplevel(1)\n    oLineData = oLineData.reset_index()[['gameId','playId','olineMinY','olineMaxY','olineMinX','olineMaxX']]\n    \n    puntPlays = puntPlays.merge(oLineData, on=['gameId','playId'])\n    \n    # Personal Protector is the OFF player who is furthest back from LOS other than the punter\n    # (They usually line up 3-5yrds off the line, but this heuristic did not hold when I spot checked plays)\n    # @Football players I also assumed all punts have a PP\n    puntPlays.loc[(puntPlays.isOffense==True)&\\\n                ((puntPlays.x==puntPlays.olineMaxX)&(puntPlays.playDirection=='left')), \"puntPosition\"] = \"PP\"\n    puntPlays.loc[(puntPlays.isOffense==True)&\\\n                ((puntPlays.x==puntPlays.olineMinX)&(puntPlays.playDirection=='right')), \"puntPosition\"] = \"PP\"\n    \n    # Wings are the players on the edges of the OLine\n    # (They usually line up 1-2yrds off the line, but this heuristic did not hold when I spot checked plays)\n    \n    puntPlays.loc[(pd.isna(puntPlays.puntPosition))&\\\n                 ((np.abs(puntPlays.y-puntPlays.olineMaxY)<0.25)|((np.abs(puntPlays.y-puntPlays.olineMinY)<0.25)))&\\\n                 (np.abs(puntPlays.x-puntPlays.losX)>2), \"puntPosition\"] = \"W\"\n\n\n    # Lineman are anyone who doesn't have a specified role in punt formation\n    puntPlays.loc[(pd.isna(puntPlays.puntPosition))&\\\n                    (puntPlays.isOffense == True), \"puntPosition\"] = 'OL'\n    puntPlays.loc[(pd.isna(puntPlays.puntPosition))&\\\n                    (puntPlays.isOffense == False), \"puntPosition\"] = 'DL'\n\n    puntPlays.loc[(puntPlays.team == \"football\"), \"puntPosition\"] = \"football\"\n    \n    if verbose:\n        print(\"done!\")\n        \n    if allFeatures:\n        return(puntPlays)\n    else: \n        return(puntPlays[['gameId','playId','nflId','isOffense','losX','puntPosition']])","metadata":{"execution":{"iopub.status.busy":"2021-11-18T21:44:06.441724Z","iopub.execute_input":"2021-11-18T21:44:06.44215Z","iopub.status.idle":"2021-11-18T21:44:06.462329Z","shell.execute_reply.started":"2021-11-18T21:44:06.442119Z","shell.execute_reply":"2021-11-18T21:44:06.461222Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"trackingData20 = pd.read_csv(\"/kaggle/input/nfl-big-data-bowl-2022/tracking2020.csv\")\nplayData = pd.read_csv('/kaggle/input/nfl-big-data-bowl-2022/plays.csv')\ngameData = pd.read_csv(\"/kaggle/input/nfl-big-data-bowl-2022/games.csv\")[['gameId','homeTeamAbbr']]","metadata":{"execution":{"iopub.status.busy":"2021-11-18T21:44:06.464067Z","iopub.execute_input":"2021-11-18T21:44:06.464895Z","iopub.status.idle":"2021-11-18T21:44:40.226216Z","shell.execute_reply.started":"2021-11-18T21:44:06.464854Z","shell.execute_reply":"2021-11-18T21:44:40.225321Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"puntPlayPd = generatePuntFeatures(trackingData20,playData,gameData)","metadata":{"execution":{"iopub.status.busy":"2021-11-18T21:44:40.228081Z","iopub.execute_input":"2021-11-18T21:44:40.22856Z","iopub.status.idle":"2021-11-18T21:44:47.50471Z","shell.execute_reply.started":"2021-11-18T21:44:40.228446Z","shell.execute_reply":"2021-11-18T21:44:47.503462Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"puntPlayPd.head()","metadata":{"execution":{"iopub.status.busy":"2021-11-18T21:44:47.506262Z","iopub.execute_input":"2021-11-18T21:44:47.506524Z","iopub.status.idle":"2021-11-18T21:44:47.528355Z","shell.execute_reply.started":"2021-11-18T21:44:47.50649Z","shell.execute_reply":"2021-11-18T21:44:47.527283Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Known Issues:**\n- When the Personal Protector (PP) is (_incorrectly_) lined up above the Wing (W), the W and PP labels will be switched. Seen on `1` play out of the hundreds reviewed\n- When the offense is in Prevent Block Formation, Gunners (G) brought in for protection obfuscate what consistutes a wing (W) player is because wings are defined as being on the edge of the line of scrmmage. As of now it labels everyone as OL","metadata":{}},{"cell_type":"markdown","source":"## Spot-Checking Results\n\nThe following is not crucial to the feature generation, but is helpful to validate + visualize the `puntPosition` column ","metadata":{}},{"cell_type":"code","source":"%matplotlib inline\nimport matplotlib.pyplot as plt\nimport matplotlib.patches as patches","metadata":{"execution":{"iopub.status.busy":"2021-11-18T21:44:47.53027Z","iopub.execute_input":"2021-11-18T21:44:47.530554Z","iopub.status.idle":"2021-11-18T21:44:47.536234Z","shell.execute_reply.started":"2021-11-18T21:44:47.530522Z","shell.execute_reply":"2021-11-18T21:44:47.535334Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Reminder that this is only punt plays so will certaintly be smaller than full tracking data\nupdatedPuntData = puntPlayPd.merge(trackingData20,on=['gameId','playId','nflId'])","metadata":{"execution":{"iopub.status.busy":"2021-11-18T21:44:47.538035Z","iopub.execute_input":"2021-11-18T21:44:47.538575Z","iopub.status.idle":"2021-11-18T21:44:54.465334Z","shell.execute_reply.started":"2021-11-18T21:44:47.538529Z","shell.execute_reply":"2021-11-18T21:44:54.464494Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"updatedPuntData.head()","metadata":{"execution":{"iopub.status.busy":"2021-11-18T21:44:54.467411Z","iopub.execute_input":"2021-11-18T21:44:54.467698Z","iopub.status.idle":"2021-11-18T21:44:54.4963Z","shell.execute_reply.started":"2021-11-18T21:44:54.467667Z","shell.execute_reply":"2021-11-18T21:44:54.495242Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Amazing function written by ANDIKARACHMAN for DataBowl 2021\n# (https://www.kaggle.com/ar2017/nfl-big-data-bowl-2021-animating-players-movement)\ndef createFootballField(linenumbers=True,\n                          endzones=True,\n                          highlight_line=False,\n                          highlight_line_number=55,\n                          highlight_first_down_line=False,\n                          yards_to_go=10,\n                          highlighted_name='Line of Scrimmage',\n                          fifty_is_los=False,\n                          figsize=(24, 12.66)):\n    \"\"\"\n    Function that plots the football field for viewing plays.\n    Allows for showing or hiding endzones.\n    \"\"\"\n    rect = patches.Rectangle((0, 0), 120, 53.3, linewidth=0.1,\n                             edgecolor='r', facecolor='darkgreen', zorder=0)\n\n    fig, ax = plt.subplots(1, figsize=figsize)\n    ax.add_patch(rect)\n\n    plt.plot([10, 10, 10, 20, 20, 30, 30, 40, 40, 50, 50, 60, 60, 70, 70, 80,\n              80, 90, 90, 100, 100, 110, 110, 120, 0, 0, 120, 120],\n             [0, 0, 53.3, 53.3, 0, 0, 53.3, 53.3, 0, 0, 53.3, 53.3, 0, 0, 53.3,\n              53.3, 0, 0, 53.3, 53.3, 0, 0, 53.3, 53.3, 53.3, 0, 0, 53.3],\n             color='white')\n    if fifty_is_los:\n        plt.plot([60, 60], [0, 53.3], color='gold')\n        plt.text(62, 50, '<- Player Yardline at Snap', color='gold')\n    # Endzones\n    if endzones:\n        ez1 = patches.Rectangle((0, 0), 10, 53.3,\n                                linewidth=0.1,\n                                edgecolor='r',\n                                facecolor='blue',\n                                alpha=0.2,\n                                zorder=0)\n        ez2 = patches.Rectangle((110, 0), 120, 53.3,\n                                linewidth=0.1,\n                                edgecolor='r',\n                                facecolor='blue',\n                                alpha=0.2,\n                                zorder=0)\n        ax.add_patch(ez1)\n        ax.add_patch(ez2)\n    plt.xlim(0, 120)\n    plt.ylim(-5, 58.3)\n    plt.axis('off')\n    if linenumbers:\n        for x in range(20, 110, 10):\n            numb = x\n            if x > 50:\n                numb = 120 - x\n            plt.text(x, 5, str(numb - 10),\n                     horizontalalignment='center',\n                     fontsize=20,  # fontname='Arial',\n                     color='white')\n            plt.text(x - 0.95, 53.3 - 5, str(numb - 10),\n                     horizontalalignment='center',\n                     fontsize=20,  # fontname='Arial',\n                     color='white', rotation=180)\n    if endzones:\n        hash_range = range(11, 110)\n    else:\n        hash_range = range(1, 120)\n\n    for x in hash_range:\n        ax.plot([x, x], [0.4, 0.7], color='white')\n        ax.plot([x, x], [53.0, 52.5], color='white')\n        ax.plot([x, x], [22.91, 23.57], color='white')\n        ax.plot([x, x], [29.73, 30.39], color='white')\n\n    if highlight_line:\n        hl = highlight_line_number + 10\n        plt.plot([hl, hl], [0, 53.3], color='yellow')\n        \n    if highlight_first_down_line:\n        fl = hl + yards_to_go\n        plt.plot([fl, fl], [0, 53.3], color='yellow')\n    return fig, ax","metadata":{"execution":{"iopub.status.busy":"2021-11-18T21:44:54.498021Z","iopub.execute_input":"2021-11-18T21:44:54.498385Z","iopub.status.idle":"2021-11-18T21:44:54.516747Z","shell.execute_reply.started":"2021-11-18T21:44:54.498342Z","shell.execute_reply":"2021-11-18T21:44:54.516022Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# i:\n    # updatedPuntData || player-tracking-level data \n    # playData || play-level data \n    # gameId || (optional) specific game to track (int) \n    # playId || (optional) specific play to visualize (int)\n# o:\n    # none. Visual of either a random play or a specified play\ndef spotCheckLabels(updatedPuntData, playData, gameId=None, playId=None):\n      \n    if (gameId != None) and (playId != None):\n        updatedPuntData.query(f\"gameId == {gameId} and playId == {playId}\")\n        \n        spotCheckPlay = playData.query(f\"gameId == {gameId} and playId == {playId}\")\n        \n        if len(spotCheckPlay) == 0:\n            print(\"Invalid GameId/playId combination\")\n            return\n        \n    else:\n\n        gamePlay = updatedPuntData.sample()[['gameId','playId']].values[0]\n        gameId = gamePlay[0]\n        playId = gamePlay[1]\n\n        spotCheckPlay = playData.query(f\"gameId == {gameId} and playId == {playId}\")\n\n    \n    spotCheckData = updatedPuntData.query(f\"gameId == {gameId} and playId == {playId} and frameId == 1\")\n    \n    yardlineNumber = spotCheckPlay['yardlineNumber'].item()\n    yardsToGo = spotCheckPlay['yardsToGo'].item()\n    absoluteYardlineNumber = spotCheckPlay['absoluteYardlineNumber'].item() - 10\n    playDesc = spotCheckPlay['playDescription'].item()\n    \n    playDir = spotCheckData.sample(1)['playDirection'].item()\n    \n    if (absoluteYardlineNumber > 50):\n        yardlineNumber = 100 - yardlineNumber\n    if (absoluteYardlineNumber <= 50):\n        yardlineNumber = yardlineNumber\n        \n    if (playDir == 'left'):\n        yardsToGo = -yardsToGo\n    else:\n        yardsToGo = yardsToGo\n    \n    fig, ax = createFootballField(highlight_line=True, highlight_line_number=yardlineNumber, highlight_first_down_line=True, yards_to_go=yardsToGo)\n    \n    plt.title(f'Game # {gameId} Play # {playId} \\n {playDesc}');\n\n    \n    playOff = spotCheckData[spotCheckData.isOffense==True]\n    playDef = spotCheckData[spotCheckData.isOffense==False]\n    playFootball = spotCheckData[spotCheckData.team==\"football\"]\n    \n    patch = []\n    offX = playOff['x']\n    offY = playOff['y']\n    offPos = playOff['puntPosition']\n    patch.extend(plt.plot(offX, offY, 'o',c='gold', ms=20, mec='white'))\n    \n    # Home players' position\n    for x, y, pos in zip(offX, offY, offPos):\n        patch.append(plt.text(x, y, pos, va='center', ha='center', color='black', size='medium'))\n        \n        \n    defX = playDef['x']\n    defY = playDef['y']\n    defPos = playDef['puntPosition']\n    patch.extend(plt.plot(defX, defY, 'o',c='orangered', ms=20, mec='white'))\n        \n    # Away players' position\n    for x, y, pos in zip(defX, defY, defPos):\n        patch.append(plt.text(x, y, pos, va='center', ha='center', color='white', size='medium'))","metadata":{"execution":{"iopub.status.busy":"2021-11-18T21:45:55.659387Z","iopub.execute_input":"2021-11-18T21:45:55.659678Z","iopub.status.idle":"2021-11-18T21:45:55.673533Z","shell.execute_reply.started":"2021-11-18T21:45:55.659649Z","shell.execute_reply":"2021-11-18T21:45:55.672283Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"spotCheckLabels(updatedPuntData, playData)","metadata":{"execution":{"iopub.status.busy":"2021-11-18T21:57:02.361153Z","iopub.execute_input":"2021-11-18T21:57:02.361403Z","iopub.status.idle":"2021-11-18T21:57:03.313761Z","shell.execute_reply.started":"2021-11-18T21:57:02.361375Z","shell.execute_reply":"2021-11-18T21:57:03.312394Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Hope this helps! Happy DataBowl-ing!","metadata":{}}]}