{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport math\nfrom pandas.core.groupby.generic import ScalarResult\nfrom sklearn.linear_model import LogisticRegression, SGDRegressor\nfrom sklearn.preprocessing import normalize\nfrom sklearn.neural_network import MLPRegressor\nimport warnings\nfrom plotly.offline import init_notebook_mode, iplot, plot\nimport plotly as py\nfrom plotly.subplots import make_subplots\ninit_notebook_mode(connected=True)\nimport plotly.graph_objs as go\n\n# 1.0 for clean catch, 0.7 for clean field, 0.5 for bobble, 0 for muff, None for Touchback, OOB, etc\ncontact_mappings = {'CC': 1.0, 'CFFG': 0.9, 'BB': 0.72, 'BC': 0.57, 'BF': 0.77, 'BOG': 0.52, 'ICC': 0.25, 'MBC': 0.0, 'MBDR': 0.0}\npd.set_option('display.max_columns', None)","metadata":{"_uuid":"01e1000b-5225-43fe-bac8-6f0467c57dae","_cell_guid":"7d0e584b-73d3-4a46-83c1-8f1149734497","jupyter":{"outputs_hidden":false},"collapsed":false,"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-05T05:17:54.942623Z","iopub.execute_input":"2022-01-05T05:17:54.943062Z","iopub.status.idle":"2022-01-05T05:17:56.366195Z","shell.execute_reply.started":"2022-01-05T05:17:54.943027Z","shell.execute_reply":"2022-01-05T05:17:56.365178Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Modeling Expected Punt Return Yards\nAlthough perhaps not as glamorous as the offensive phase or as schematically complicated as the defensive phase, the punt is one of the most pivotal plays in the game of football.  Bridging the gap between the two main phases of the game, a good punt can set a defense up for success by pinning the opposing team deep in their own territory, shrinking the playbook down to low-risk, low-reward plays and increasing the odds of getting the ball back.  Conversely, a good punt return can set an offense up for success.  The game of football is a balance of scheme and execution, technique and athleticism.  What makes a punt and corresponding return so special is that in no other phase of the game does that balance shift more towards instinct and natural ability.  While there are certain basic strategies a unit may take, the play essentially boils down to a runner and 10 blockers versus 11 defenders trying to tackle the runner.  This lack of diversity in strategy has made quantifying and projecting punt/punt return performance traditionally difficult.  However, with the tracking data provided by the competition, we will attempt to build a new metric, **Expected Punt Return Yards**, to evaluate returner performance using *Punt Return Yards Over Expected* (**PRYOE**), and conversely to evaluate punt coverage performance using *Punt Return Yards Over Expected Allowed* (**PRYOEA**).  ","metadata":{}},{"cell_type":"code","source":"def read_inputs():\n    ## Read input data\n    scouting_df = pd.read_csv('../input/nfl-big-data-bowl-2022/PFFScoutingData.csv')\n    games_df = pd.read_csv('../input/nfl-big-data-bowl-2022/games.csv')\n    plays_df = pd.read_csv('../input/nfl-big-data-bowl-2022/plays.csv')\n    tracking_df = pd.DataFrame()\n    tracking_2018 = pd.read_csv('../input/nfl-big-data-bowl-2022/tracking2018.csv')\n    tracking_2019 = pd.read_csv('../input/nfl-big-data-bowl-2022/tracking2019.csv')\n    tracking_2020 = pd.read_csv('../input/nfl-big-data-bowl-2022/tracking2020.csv')\n    tracking_df = tracking_df.append(tracking_2018)\n    tracking_df = tracking_df.append(tracking_2019)\n    tracking_df = tracking_df.append(tracking_2020)\n    players_df = pd.read_csv('../input/nfl-big-data-bowl-2022/players.csv')\n    return scouting_df, games_df, plays_df, tracking_df, players_df\nscouting_df, games_df, plays_df, tracking_df, players_df = read_inputs()","metadata":{"_uuid":"0b42163b-24c3-490b-bd7f-a86a10408d1a","_cell_guid":"89a3ed30-1005-4ec5-b0e6-58f6e66dcfcd","collapsed":false,"jupyter":{"outputs_hidden":false},"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-05T05:17:56.368071Z","iopub.execute_input":"2022-01-05T05:17:56.368516Z","iopub.status.idle":"2022-01-05T05:20:57.717433Z","shell.execute_reply.started":"2022-01-05T05:17:56.36847Z","shell.execute_reply":"2022-01-05T05:20:57.715446Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def distanceBetween(x1, y1, x2, y2):\n    a = np.array((x1, y1))\n    b = np.array((x2, y2))\n    c = np.linalg.norm(a-b)\n    return c\n\ndef getFrameIDOfPuntCatch(tracking_df, plays_df):\n    plays_df['season'] = plays_df.apply(lambda x: int(str(x['gameId'])[:4]), axis=1)\n    punts_df = plays_df[['gameId', 'playId', 'season', 'specialTeamsPlayType', 'returnerId']].query('specialTeamsPlayType == \"Punt\"')\n    temp_df = tracking_df[['event', 'gameId', 'playId', 'frameId']]\n    temp_df = temp_df.query('event == \"kick_received\" | event == \"punt_land\" | event == \"punt_downed\" | event == \"punt_received\" | event == \"punt_muffed\" | event == \"fair_catch\"')\n    temp_df.set_index(['gameId', 'playId'], inplace=True)\n    punts_df.set_index(['gameId', 'playId'], inplace=True)\n    catch_df = temp_df.join(punts_df, on=['gameId', 'playId'], how='inner')\n    catch_df = catch_df.drop_duplicates()\n    catch_df.head()\n    return catch_df\n\ndef getDistancesAtTimeOfCatch(catch_df, tracking_df):\n    catch_df.reset_index(inplace=True)\n    \n    catch_df.dropna(subset=['returnerId'], axis=0, inplace=True)\n    \n    tracking_df.set_index(['gameId', 'playId', 'frameId'], inplace=True)\n    catch_df.set_index(['gameId', 'playId', 'frameId'], inplace=True)\n    \n    full_df = catch_df.join(tracking_df, on=['gameId', 'playId', 'frameId'], how='inner', rsuffix='r_')\n    \n    full_df.reset_index(inplace=True)\n    catch_df.reset_index(inplace=True)\n    \n    punts_returners = catch_df[['gameId', 'playId', 'returnerId']].values.tolist()\n    \n    col_arr = []\n    yd_arr = []\n    in_radius_arr = []\n    for punt in punts_returners:\n        play = full_df.query('gameId == @punt[0] & playId == @punt[1]').dropna(subset=['nflId'])\n        returner_id = int(punt[2].split(';')[0])\n        returner_loc = play.query('nflId == @returner_id')[['x','y', 'team']].values.tolist()\n        closest_loc = 1000.0\n        play_in_rad = 0\n        for row in play.itertuples():\n            if int(row.nflId) == returner_id or row.team == returner_loc[0][2]:\n                pass\n            else:\n                this_loc = distanceBetween(returner_loc[0][0], returner_loc[0][1], row.x, row.y)\n                if this_loc < closest_loc:\n                    closest_loc = this_loc\n                if this_loc < 8.0:\n                    play_in_rad += 1\n        col_arr.append(closest_loc)\n        yd_arr.append(returner_loc[0][1])\n        in_radius_arr.append(play_in_rad)\n        \n    col = pd.Series(data=col_arr)\n    yd = pd.Series(data=yd_arr)\n    in_rad = pd.Series(data=in_radius_arr)\n    yd = yd.apply(lambda x: 0 if (x < 10.0 or x > 110.0) else (int(120.0 - x) if x > 60.0 else int(x)))\n    catch_df['closest_gunner'] = col\n    catch_df['catch_yardline'] = yd\n    catch_df['defenders_within_radius'] = in_rad\n    catch_df.drop(['frameId', 'season', 'specialTeamsPlayType', 'returnerId'], axis=1, inplace=True)\n    \n    return catch_df\n\ndef getReturnerAvg(df):\n    df = df.copy()[['returnerId', 'kickReturnYardage']]\n    returner_df = df.groupby(['returnerId']).mean()\n    returner_df['returnAvg'] = returner_df['kickReturnYardage']\n    returner_df.drop(['kickReturnYardage'], axis=1, inplace=True)\n    return returner_df","metadata":{"_uuid":"d98a1709-405f-47ea-afbd-d153cb416cf4","_cell_guid":"5a26e334-dbd1-4a89-b4c7-5a85b802c158","collapsed":false,"jupyter":{"outputs_hidden":false},"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-05T05:20:57.719869Z","iopub.execute_input":"2022-01-05T05:20:57.720218Z","iopub.status.idle":"2022-01-05T05:20:57.74949Z","shell.execute_reply.started":"2022-01-05T05:20:57.72018Z","shell.execute_reply":"2022-01-05T05:20:57.747952Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Feature Extraction\n\nIn order to build a successful model, we first need to consider which aspects of a play we can measure, and which may be impactful to a punt return.  \n\n## Dataset Features\nAfter examining the dataset, there are a number of measurements that jump out as immediately applicable.  These include:\n* *Kick Length*\n* *Punt Hangtime*\n* *Number of Gunners (Punt team players that sprint down the field to tackle returner)*\n* *Number of Punt Safeties (Punt team players that hang back)*\n* *Number of Vises (Return team players that line up opposite gunners and prevent them from tackling returner)*\n* *Number of Punt Rushers (Return team players that rush the punter and attempt to block the punt)*\n\n## Encoded Features\nThere are also several columns that, given some encoding, can be easily quantified.  These include:\n* *Kick Direction (intended and actual)*\n    * This is encoded as the difference between the two.  E.g. (intended left, actual left) = 1.0, (intended left, actual center) = 0.5, (intended left, actual right) = 0.0.\n* *Return Direction (intended and actual)*\n    * Same scale as above.\n* *Catch Quality*\n    * Encoded on a range scale: 1.0 for clean catch, 0.7 for clean field, 0.5 for bobble, 0 for muff.\n* *Snap Quality*\n    * Binary encoding, 1 if snap quality was good ('OK' in dataset), and 0 if it was anything else. \n\n## Derived Features\nFinally, using the tracking data, we can build some more complex features that give us insight into what happens during the play.  These include:\n* *Yardline where the punt is received*\n    * From perspective of leaving your own endzone.\n* *Returner's past average return yards*\n* *How close the nearest punt team player is to returner at time of catch*\n* *Number of punt team players within returners **Radius of Concern** at time of catch*\n    * For our purposes, **RoC** set to 8 yards.  With more time, this number could be optimized further.\n\n## Truth Features\nIn order to model punt return yards, we need to identify both that a punt has been returned and how many yards were gained on the return.  Therefore our *Ground Truth* features are:\n* *Was the punt returned?*\n    * 1 for returned, 0 for not returned.\n* *How many yards were gained on the return?*\n    * Difference between yardline of catch and yardline of tackle.\n\nIn the case of our model, we are not concerned with non-routine play outcomes.  Events like blocked punts, fumbled snaps, fake punts, etc. are not returnable plays, and are therefore irrelevant to our investigation.  These plays have been removed from our feature set.  ","metadata":{}},{"cell_type":"code","source":"def encode_columns(plays_df, scouting_df, tracking_df, games_df):\n    ## Preprocessing step for encoding inputs\n    games_df = games_df.copy()\n    games_df = games_df.reset_index(drop=True)\n    \n    # Get playIDs for all punts, as well as other raw columns that require no further manipulation\n    params_df = plays_df[['gameId', 'playId', 'specialTeamsPlayType', 'kickerId', 'returnerId', 'kickLength', 'kickReturnYardage', 'possessionTeam']].query('specialTeamsPlayType == \"Punt\"')\n    scouting_df = scouting_df[['gameId', 'playId', 'snapDetail', 'operationTime', 'hangTime', 'kickType', 'kickDirectionIntended', 'kickDirectionActual', 'returnDirectionIntended', 'returnDirectionActual', 'gunners', 'puntRushers', 'specialTeamsSafeties', 'vises', 'kickContactType']]\n    params_df.set_index(['gameId', 'playId'], inplace=True)\n    scouting_df.set_index(['gameId', 'playId'], inplace=True)\n    params_df = params_df.join(scouting_df, on=['gameId', 'playId'])\n    games_df = games_df.reset_index()\n    params_df = params_df.reset_index()\n    \n    # Non-inline Columns\n    catch_df = getFrameIDOfPuntCatch(tracking_df, plays_df)\n    temp_df = getDistancesAtTimeOfCatch(catch_df.copy(), tracking_df)\n    temp_df.set_index(['gameId', 'playId'], inplace=True)\n    \n    # Derived Columns\n    params_df['season'] = params_df['gameId'].apply(lambda x: games_df.loc[x == games_df['gameId'], ['season']].values[0][0])\n    params_df.set_index(['gameId', 'playId'], inplace=True)\n    params_df['snapQuality'] = params_df['snapDetail'].apply(lambda x: 1 if x == 'OK' else 0)\n    params_df['kickDirDiff'] = params_df[['kickDirectionIntended', 'kickDirectionActual']].apply(lambda x: 1 if x[0] == x['kickDirectionActual'] else (0.5 if (x['kickDirectionIntended'] == 'C' or x['kickDirectionActual'] == 'C') else 0), axis=1)\n    params_df['retDirDiff'] = params_df[['returnDirectionIntended', 'returnDirectionActual']].apply(lambda x: 1 if x['returnDirectionIntended'] == x['returnDirectionActual'] else (0.5 if (x['returnDirectionIntended'] == 'C' or x['returnDirectionActual'] == 'C') else 0), axis=1)\n    params_df['numGunners'] = params_df['gunners'].apply(lambda x: 0 if pd.isna(x) else (len(x.split(';'))))\n    params_df['numVises'] = params_df['vises'].apply(lambda x: 0 if pd.isna(x) else (len(x.split(';'))))\n    params_df['numSafeties'] = params_df['specialTeamsSafeties'].apply(lambda x: 0 if pd.isna(x) else (len(x.split(';'))))\n    params_df['numRushers'] = params_df['puntRushers'].apply(lambda x: 0 if pd.isna(x) else (len(x.split(';'))))\n    params_df['catchQuality'] = params_df['kickContactType'].apply(lambda x: contact_mappings[x] if x in contact_mappings.keys() else None)\n    params_df['kickReturnYardage'].fillna(0, inplace=True)\n    params_df['catchQuality'].fillna(0.0, inplace=True)\n    returner_df = getReturnerAvg(params_df)\n    params_df = params_df.join(returner_df, on=['returnerId'], how='left')\n    params_df = params_df.join(temp_df, on=['gameId', 'playId'], how='inner')\n    params_df['isReturn'] = params_df[['event', 'kickReturnYardage', 'catchQuality']].apply(lambda x: 0 if (x['event'] == 'fair_catch' or x['event'] == 'punt_downed' or (x['kickReturnYardage'] == 0 and x['event'] == 'punt_land' and x['catchQuality'] == 0)) else 1, axis=1)\n    \n    # Remove non-feature columns and re-index data\n    params_df.drop(['gunners', 'vises', 'specialTeamsSafeties', 'puntRushers', 'returnDirectionIntended', 'returnDirectionActual', 'kickDirectionIntended', 'kickDirectionActual', 'snapDetail', 'kickType', 'kickContactType'], axis=1, inplace=True)\n    params_df.reset_index(inplace=True)\n    \n    return params_df, returner_df\nparams_df, returners_df = encode_columns(plays_df, scouting_df, tracking_df, games_df)\nparams_df.head()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-05T05:20:57.751325Z","iopub.execute_input":"2022-01-05T05:20:57.75164Z","iopub.status.idle":"2022-01-05T05:28:13.057793Z","shell.execute_reply.started":"2022-01-05T05:20:57.751563Z","shell.execute_reply":"2022-01-05T05:28:13.056328Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Building a Model\nBefore attempting to mathematically model a punt return, we must first consider what goes through the punt returners head on a given play.  After the ball is punted, the punt returner has 3 options:\n* Do nothing and let the ball land, allowing the punting team to \"down\" the ball wherever it may rest\n* Catch the ball but not return it, either by a fair catch or taking a knee\n* Catch the ball and return it\n\nIn the case of the first two outcomes, there is no return.  This allows us to reduce the initial step to a binary decision: *Should the punt be returned or not returned?*\n\nFrom there, the next question that must be answered is: *Given that the punt is returned, how many yards do we expect the returner to gain?*\n\n## Design\nNow that we have identified the questions that must be answered, we can determine the optimal techniques for answering each.  \n\nThe first question is a yes/no question.  This lends itself to using a *Classification* technique.  Candidates for this include:\n* Logistic Regression\n* Ensemble Classification (e.g. Random Forest Classifier)\n* Multilayer Perceptron (Neural Net) Binary Classifier\n\nAfter testing these three candidate techniques, the most accurate model was using a Logistic Regression with Newton's Method as the specified solver function.  \n\nThe second question requires a quantity as an answer.  This lends itself to using a *Regression* technique.  Candidates for this include:\n* Linear Regression\n* Gradient Descent Regression\n* Multilayer Perceptron (Neural Net) Regressor\n\nAfter testing these model candidates, the most accurate turned out to be a variation of a Gradient Descent Regressor, a *Stochastic Gradient Descent* model.  *SGD* is essentially a Gradient Descent Regressor that uses a random subset of data to calculate error on each iteration instead of the entire dataset.  ","metadata":{}},{"cell_type":"markdown","source":"## Testing\nOur full, cleaned dataset includes about 3200 punts/returns from the 2018-2020 seasons.  We will reserve 200 plays for testing, and use the rest for training.\n\n### Return Model\nA typical Logistic Regression model returns a probability matrix, where the option with >50% probability is selected as the classification output.  However, when testing our model, it was discovered that moving that threshold to 60% probability resulted in significantly higher accuracy.  The breakdown is as follows:","metadata":{}},{"cell_type":"code","source":"def test_logistic_model(df):\n    df = df.copy()\n    outcomes = ['True Positive', 'False Positive', 'True Negative', 'False Negative']\n    t_p = 0\n    f_p = 0\n    t_n = 0\n    f_n = 0\n    df.drop(['gameId', 'playId', 'event', 'specialTeamsPlayType', 'kickerId', 'returnerId', 'kickReturnYardage', 'possessionTeam', 'season'], axis=1, inplace=True)\n    model = LogisticRegression(solver='newton-cg')\n    df = df.sample(frac=1).reset_index(drop=True)\n    y = df['isReturn'].to_numpy()\n    X = df.drop(['isReturn'], axis=1)\n    X.fillna(0.0, inplace=True)\n    X = normalize(X, axis=0)\n    Xone = X[:-200]\n    yone = y[:-200]\n    Xtwo = X[-199:]\n    ytwo = y[-199:]\n\n    model.fit(Xone, yone)\n    total = len(ytwo)\n    correct = 0\n    for i in range(len(ytwo)):\n        yres = model.predict([Xtwo[i]])\n        confidence = model.predict_proba([Xtwo[i]])\n        manual = 0\n        if confidence[0][1] > 0.6 and ytwo[i] == 1:\n            t_p += 1\n        elif confidence[0][1] > 0.6 and ytwo[i] == 0:\n            f_p += 1\n        elif confidence[0][1] < 0.6 and ytwo[i] == 1:\n            f_n += 1\n        elif confidence[0][1] < 0.6 and ytwo[i] == 0:\n            t_n += 1\n        if confidence[0][1] > 0.6:\n            manual = 1\n        if manual == ytwo[i]:\n            correct += 1\n    fig = make_subplots(rows=1, cols=2, specs=[[{'type': 'xy'}, {'type': 'domain'}]], subplot_titles=['Confusion Distribution', 'Accuracy Breakdown'])\n    fig.add_trace(go.Bar(x = outcomes, y = [t_p, f_p, t_n, f_n], name=\"Specific Results\", showlegend=False), 1,1)\n    fig.layout['yaxis'].update(title_text=\"Count\")\n    fig.add_trace(go.Pie(labels = ['Correct', 'Incorrect'], values = [t_p + t_n, f_p + f_n], name=\"Overall Results\"), 1,2)\n    fig.show()\n    manual_score = correct/total\n    print(\"manual score: {}\".format(manual_score))\ntest_logistic_model(params_df)","metadata":{"_uuid":"4944c24b-7aec-4edb-8330-d219961ced26","_cell_guid":"395076f1-0b42-4d47-b074-4a3d412c9441","collapsed":false,"jupyter":{"outputs_hidden":false},"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-05T05:28:13.060835Z","iopub.execute_input":"2022-01-05T05:28:13.061133Z","iopub.status.idle":"2022-01-05T05:28:13.559045Z","shell.execute_reply.started":"2022-01-05T05:28:13.061102Z","shell.execute_reply":"2022-01-05T05:28:13.55767Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This model ends up giving us an accuracy of between 80% and 90%.  ","metadata":{}},{"cell_type":"markdown","source":"## Yardage Model\nWith the yardage model, evaluation becomes slightly more difficult.  An error of 2 when the prediction is 2 yards and the actual is 4 yards seems to be a miss, whereas an error of 2 when the prediction is 25 yards and the actual is 27 yards seems to be fairly accurate.  To be consistent with scoring from our binary return model, we also need our score to be in the range [0,1] (inclusive).  To account for this, we will score using the equation: \n\n$$score = \\dfrac{(1 - \\left| \\dfrac{\\dfrac{y_{_{pred}}}{2} + y_{_{pred}}}{y_{act}} - \\dfrac{\\dfrac{y_{_{pred}}}{2}+y_{act}}{y_{act}} \\right|) + 1}{\\max(y_{{pred}}, y_{act})}$$\n\nThis may seem overly complex, but allows us to scale the error on a curve according to the magnitude of projection and truth, while accounting for both positive and negative predictions and actual values.  Given more time, a more concise scoring function could be designed, which would allow for more accurate evaluation.  ","metadata":{}},{"cell_type":"code","source":"def test_linear_model(df):\n    with warnings.catch_warnings():\n        warnings.simplefilter(\"ignore\")\n        df = df.copy()\n        df = df.query('isReturn == 1')\n        df.drop(['gameId', 'playId', 'event', 'specialTeamsPlayType', 'kickerId', 'returnerId', 'isReturn', 'possessionTeam', 'season'], axis=1, inplace=True)\n        model = SGDRegressor(alpha=0.000002, penalty='elasticnet', max_iter=3000)\n\n        y = df['kickReturnYardage'].to_numpy()\n        X = df.drop(['kickReturnYardage'], axis=1)\n        X.fillna(0.0, inplace=True)\n\n        X = normalize(X, axis=0) \n\n        Xone = X[:-200]\n        yone = y[:-200]\n        Xtwo = X[-199:]\n        ytwo = y[-199:]\n        model.fit(Xone, yone)\n        ypred = []\n        yact = []\n        yscore = []\n        ycorr = 0\n        yinc = 0\n        for i in range(len(ytwo)):\n            pred = model.predict([Xtwo[i]])[0]\n            act = ytwo[i]\n            if act == 0:\n                act = 0.1\n            yards_score = (1.0 - abs(((pred/2 + int(pred)) / pred) - ((pred/2 + int(act)) / pred))) + 1 / (max(pred, act))\n            if yards_score < 0:\n                yards_score = 0\n            if yards_score > 0.8:\n                ycorr += 1\n            else:\n                yinc += 1\n            if yards_score > 1.0:\n                yards_score=1\n            ypred.append(pred)\n            yact.append(act)\n            yscore.append(yards_score)\n        fig = go.Figure(go.Scatter(x=yact, y=ypred, name='Score', mode='markers', hovertemplate=\"<b>Pred. Yards</b>: %{y:.2f}<br><b>Act. Yards</b>: %{x:.2f}\", marker=dict(color=yscore, colorscale='rdbu_r', showscale=True)))\n        fig.layout['xaxis'].update(title_text='Actual Yards')\n        fig.layout['yaxis'].update(title_text='Expected Yards')\n        fig.update_layout(title_text=\"Actual Yards vs Expected Yards\", title_x=0.5, title_xanchor='center', legend_title=\"Score\")\n        fig.show()\n        score = model.score(Xtwo, ytwo)\n        print(score)\n        print('% in acceptable range: {}%'.format((ycorr / (ycorr + yinc)) * 100))\ntest_linear_model(params_df)","metadata":{"_uuid":"aa8f9d16-ce0a-4029-a2b1-56c399725107","_cell_guid":"d64c368c-9786-465e-9e37-6cf0277b6371","collapsed":false,"jupyter":{"outputs_hidden":false},"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-05T05:28:13.568787Z","iopub.execute_input":"2022-01-05T05:28:13.569087Z","iopub.status.idle":"2022-01-05T05:28:14.114051Z","shell.execute_reply.started":"2022-01-05T05:28:13.569055Z","shell.execute_reply":"2022-01-05T05:28:14.112673Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As you can see, the model struggles with returns at either extreme.  However, it is excellent at predicting yardage in the 3-10 yard range, which is where most \"expected\" returns are.  The outliers greater than 15 yards and less than 1 yard are examples of returns where a returner had significantly more or less **YoE**, respectively.  ","metadata":{}},{"cell_type":"code","source":"def fit_logistic_model(df):\n    df = df.copy()\n    df.drop(['gameId', 'playId', 'event', 'specialTeamsPlayType', 'kickerId', 'returnerId', 'kickReturnYardage', 'possessionTeam', 'season'], axis=1, inplace=True)\n    model = LogisticRegression(solver='newton-cg')\n    df = df.sample(frac=1).reset_index(drop=True)\n    y = df['isReturn'].to_numpy()\n    X = df.drop(['isReturn'], axis=1)\n    X.fillna(0.0, inplace=True)\n    X = normalize(X, axis=0)\n    model.fit(X, y)\n    return model\n\ndef fit_linear_model(df):\n    df = df.copy()\n    df.drop(['gameId', 'playId', 'event', 'specialTeamsPlayType', 'kickerId', 'returnerId', 'isReturn', 'possessionTeam', 'season'], axis=1, inplace=True)\n    model = SGDRegressor(alpha=0.00002, penalty='elasticnet', max_iter=10000, early_stopping=True)\n    \n    y = df['kickReturnYardage'].to_numpy()\n    X = df.drop(['kickReturnYardage'], axis=1)\n    X.fillna(0.0, inplace=True)\n    \n    X = normalize(X, axis=0)    \n        \n    model.fit(X, y)\n    return model","metadata":{"_uuid":"a090fdd8-bae6-41f8-ac4a-cdfa37b74efe","_cell_guid":"9b89408d-b5b6-41ef-a324-6632ab621c42","collapsed":false,"jupyter":{"outputs_hidden":false},"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-05T05:28:14.11572Z","iopub.execute_input":"2022-01-05T05:28:14.116142Z","iopub.status.idle":"2022-01-05T05:28:14.129749Z","shell.execute_reply.started":"2022-01-05T05:28:14.115992Z","shell.execute_reply":"2022-01-05T05:28:14.128143Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Two-Step Model\nNow that we have built and tested both of our models, it is time to put them together.  We will first pass each punt into our binary return model.  We can evaluate according to the following four scenarios:\n* We predict *no return* and we get *no return*: We guessed correct, and can give ourselves a score of **1**\n* We predict *return* and we get *no return*: We guessed wrong and can give ourselves a score of **0**\n* We predict *no return* and we get *return*: We guessed wrong and can give ourselves a score of **0**\n* We predict *return* and get *return*: We guessed correct, and will assign ourselves a score between **0 and 1** according to our above scoring function","metadata":{}},{"cell_type":"code","source":"def linear_two_step(df):\n    df = df.copy()\n    scores = []\n    df = df.sample(frac=1).reset_index(drop=True)\n    train_df = df.iloc[:-300]\n    test_df = df.iloc[-299:]\n    return_model = fit_logistic_model(train_df)\n    yardage_model = fit_linear_model(train_df)\n    ytest = test_df[['isReturn', 'kickReturnYardage']].to_numpy()\n    Xtest = test_df.drop(['gameId', 'playId', 'event', 'specialTeamsPlayType', 'kickerId', 'returnerId', 'isReturn', 'kickReturnYardage', 'possessionTeam', 'season'], axis=1)\n    Xtest.fillna(0.0, inplace=True)\n    Xtest = normalize(Xtest, axis=0)\n    linear_scores = []\n    ypred = []\n    yact = []\n    yscore = []\n    ycorr = 0\n    yinc = 0\n    for i in range(len(ytest)):\n        return_res = return_model.predict_proba([Xtest[i]])\n        return_manual = 0\n        if return_res[0][1] > 0.6:\n            return_manual = 1\n        if return_manual == 0 and ytest[i][0] == 0:\n            scores.append(1)\n            ypred.append(0)\n            yact.append(0)\n            yscore.append(1)\n            ycorr += 1\n        elif return_manual == 1 and ytest[i][0] == 0:\n            scores.append(0)\n            ypred.append(1)\n            yact.append(0)\n            yscore.append(0)\n            yinc += 1\n        elif return_manual == 0 and ytest[i][0] == 1:\n            scores.append(0)\n            ypred.append(0)\n            yact.append(1)\n            yscore.append(0)\n            yinc += 1\n        else:\n            yards_res = yardage_model.predict([Xtest[i]])[0]\n            if ytest[i][1] == 0:\n                ytest[i][1] = 0.1\n            yards_score = (1.0 - abs(((yards_res/2 + int(yards_res)) / yards_res) - ((yards_res/2 + int(ytest[i][1])) / yards_res))) + 1 / (max(yards_res, ytest[i][1]))\n            if yards_score < 0:\n                yards_score = 0\n            if yards_score > 0.8:\n                ycorr += 1\n            else:\n                yinc +=1\n            if yards_score > 1.0:\n                yards_score=1\n            scores.append(yards_score)\n            linear_scores.append(yards_score)\n            ypred.append(yards_res)\n            yact.append(ytest[i][1])\n            yscore.append(yards_score)\n            \n    linear_acc = sum(linear_scores) / len(linear_scores)\n    overall_acc = sum(scores) / len(scores)\n    fig = make_subplots(rows=1, cols=2, specs=[[{'type': 'xy'}, {'type': 'domain'}]], subplot_titles=['Expected Return Yards vs Actual Return Yards', 'Accuracy Breakdown'])\n    fig.add_trace(go.Scatter(x=yact, y=ypred, mode='markers', hovertemplate=\"<b>Pred. Yards</b>: %{y:.2f}<b><br>Act. Yards</b>: %{x:.2f}\", marker=dict(color=yscore, colorscale='rdbu_r', showscale=False), showlegend=False))\n    fig.layout['xaxis'].update(title_text='Actual Yards')\n    fig.layout['yaxis'].update(title_text='Expected Yards')\n    fig.add_trace(go.Pie(labels = ['Correct', 'Incorrect'], values = [ycorr, yinc], name=\"Overall Results\"), 1,2)\n    fig.show()\nlinear_two_step(params_df)","metadata":{"_uuid":"d2ad6e72-b43d-490a-bf6d-0048bd99d195","_cell_guid":"e7901bb5-5876-42c6-bcc4-5bb0a95c7b85","collapsed":false,"jupyter":{"outputs_hidden":false},"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-05T05:28:14.132047Z","iopub.execute_input":"2022-01-05T05:28:14.1324Z","iopub.status.idle":"2022-01-05T05:28:14.365436Z","shell.execute_reply.started":"2022-01-05T05:28:14.132362Z","shell.execute_reply":"2022-01-05T05:28:14.364539Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Results\n## Players\nNow that we have established a model to predict Expected Punt Return Yards (EPRY), we can compare each players expected performance versus their actual performance.  If we sum a players EPRY and subtract that from the sum of their actual punt return yards.  This gives us **Punt Return Yards Over Expected** (**PRYOE**) for each player.  Of course, this is a cumulative value that depends on the number of opportunities for each player, so we can divide by the number of returns for each player, giving us an average **PRYOE** per return.\n","metadata":{}},{"cell_type":"code","source":"def format_player_results(pryoe_df, players_df):\n    pryoe_df = pryoe_df.copy()\n    players_df = players_df.loc[:,['nflId', 'displayName']]\n    players_df.loc[:,'returnerId'] = players_df['nflId']\n    players_df = players_df.reset_index()\n    players_df = players_df.drop(['nflId'], axis=1)\n    pryoe_df.loc[:,'returnerId'] = pryoe_df['returnerId'].astype(int)\n    players_df.loc[:,'returnerId'] = players_df['returnerId'].astype(int)\n    pryoe_df.set_index(['returnerId'], inplace=True)\n    players_df.set_index(['returnerId'], inplace=True)\n    pryoe_df = pryoe_df.join(players_df, on='returnerId', how='left')\n    pryoe_df = pryoe_df.drop(['index'], axis=1)\n    return pryoe_df\n\ndef l2_normalize(col):\n    col = col.to_numpy()\n    col = col.reshape(-1,1)\n    col = normalize(col, axis=0)\n    col = col.flatten()\n    return pd.Series(col)\n\ndef evaluate_players(params_df, returners_df, return_model, yardage_model):\n    params_df = params_df.copy()\n    pyroe_df = pd.DataFrame(columns=['returnerId', 'PRYOE', 'PRYOE_AVG', 'returnerAvg', 'numReturns'])\n    headers = ['gameId', 'playId', 'event', 'specialTeamsPlayType', 'kickerId', 'returnerId', 'isReturn', 'kickReturnYardage', 'possessionTeam', 'season']\n    features = [col for col in params_df.columns if col not in headers]\n    params_df.fillna(0.0, inplace=True)\n    for feature in features:\n        params_df.loc[:,feature] = l2_normalize(params_df[feature])\n    for returner in returners_df.iterrows():\n        returnerId = returner[0]\n        if not ';' in str(returnerId):\n            returns = params_df.query('returnerId == @returnerId') \n            returner_pryoe = 0\n            returner_yds = []\n            for _return in returns.iterrows():\n                yret = _return[1]['isReturn']\n                yyrd = _return[1]['kickReturnYardage']\n                X = _return[1].drop(headers)\n                predReturn = return_model.predict_proba([X])\n                if predReturn[0][1] > 0.6:\n                    predYds = yardage_model.predict([X])[0]\n                    pryoe = yyrd - predYds\n                    returner_pryoe += pryoe\n                    returner_yds.append(yyrd)\n            try:\n                returnerAvg = sum(returner_yds) / len(returner_yds)\n                pryoeAvg = returner_pryoe / len(returner_yds)\n            except:\n                returnerAvg = 0\n                pryoeAvg = 0\n            row = {'returnerId': returnerId, 'PRYOE': returner_pryoe, 'PRYOE_AVG': pryoeAvg, 'returnerAvg': returnerAvg, 'numReturns': len(returner_yds)}\n            pyroe_df = pyroe_df.append(row, ignore_index=True)\n    pyroe_df.sort_values(by='PRYOE_AVG', axis=0, inplace=True, ascending=False, na_position='last', ignore_index=True)\n    return pyroe_df\n\nreturn_model = fit_logistic_model(params_df)\nyardage_model = fit_linear_model(params_df)\npryoe_df = evaluate_players(params_df, returners_df, return_model, yardage_model)\npryoe_df = format_player_results(pryoe_df, players_df)\npryoe_df = pryoe_df.query('numReturns > 5')\npryoe_df.head(10)","metadata":{"_uuid":"b9d0271e-804c-49e5-8700-a83cbfce3c41","_cell_guid":"0c003a7f-931a-46ae-ae4c-f01aaaad6dfb","collapsed":false,"jupyter":{"outputs_hidden":false},"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-05T05:28:14.367228Z","iopub.execute_input":"2022-01-05T05:28:14.367815Z","iopub.status.idle":"2022-01-05T05:28:18.456474Z","shell.execute_reply.started":"2022-01-05T05:28:14.367769Z","shell.execute_reply":"2022-01-05T05:28:18.455011Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"These results seem to pass the eye test, as the majority of these players are held in high esteem for their return abilities.  Now that we have established a metric for quantifying exceptional punt return performance, hopefully it can be used to identify value in players that had previously gone unnoticed.  ","metadata":{}},{"cell_type":"markdown","source":"# Units\nNow that we have identified the best returners, we can take a look at which punt coverage units have given up the least PRYOE.  Unlike punt returners, the individual players that make up a punt coverage unit change year to year, so in order to have fair comparisons we must split out teams by season.  ","metadata":{}},{"cell_type":"code","source":"def build_units_df(games_df, plays_df):\n    units_df = games_df[['gameId', 'season']]\n    plays_df = plays_df[['gameId', 'possessionTeam']]\n    \n    units_df = units_df.reset_index(drop=True)\n    plays_df = plays_df.reset_index(drop=True)\n    \n    units_df = units_df.set_index('gameId')\n    plays_df = plays_df.set_index('gameId')\n\n    units_df = plays_df.join(units_df, on='gameId', how='inner', rsuffix='_r')\n    units_df = units_df.drop_duplicates()\n    return units_df\n\ndef evaluate_units(params_df, return_model, yardage_model):\n    params_df = params_df.copy()\n    unit_pyroe_df = pd.DataFrame(columns=['Team', 'season', 'PRYOE', 'PRYOE_AVG', 'returnAvg', 'numReturns'])\n    units_df = build_units_df(games_df, plays_df)\n    headers = ['gameId', 'playId', 'event', 'specialTeamsPlayType', 'kickerId', 'returnerId', 'isReturn', 'kickReturnYardage', 'possessionTeam', 'season']\n    features = [col for col in params_df.columns if col not in headers]\n    params_df.fillna(0.0, inplace=True)\n    for feature in features:\n        params_df[feature] = l2_normalize(params_df[feature])\n    with warnings.catch_warnings():\n        warnings.simplefilter(\"ignore\")\n        for unit in units_df.iterrows():\n            unit_team = unit[1].possessionTeam\n            unit_season = unit[1].season\n            returns = params_df.query('possessionTeam == @unit_team & season == @unit_season')\n            unit_pryoe = 0\n            unit_yds = []\n            for _return in returns.iterrows():\n                yyrd = _return[1]['kickReturnYardage']\n                X = _return[1].drop(headers)\n                predReturn = return_model.predict_proba([X])\n                if predReturn[0][1] > 0.6:\n                        predYds = yardage_model.predict([X])[0]\n                        pryoe = yyrd - predYds\n                        unit_pryoe += pryoe\n                        unit_yds.append(yyrd)\n            try:\n                unitAvg = sum(unit_yds) / len(unit_yds)\n                pryoeAvg = unit_pryoe / len(unit_yds)\n            except:\n                unitAvg = 0\n                pryoeAvg = 0\n            row = {'Team': unit_team, 'season': unit_season,'PRYOE': unit_pryoe, 'PRYOE_AVG': pryoeAvg, 'returnAvg': unitAvg, 'numReturns': len(unit_yds)}\n            unit_pyroe_df = unit_pyroe_df.append(row, ignore_index=True)\n    unit_pyroe_df.sort_values(by='PRYOE_AVG', axis=0, inplace=True, ascending=True, na_position='last', ignore_index=True)\n    unit_pyroe_df.head()\n    return unit_pyroe_df\nunits_df = evaluate_units(params_df, return_model, yardage_model)\nunits_df.head(10)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-05T05:28:18.48161Z","iopub.execute_input":"2022-01-05T05:28:18.482092Z","iopub.status.idle":"2022-01-05T05:28:21.947023Z","shell.execute_reply.started":"2022-01-05T05:28:18.481939Z","shell.execute_reply":"2022-01-05T05:28:21.946005Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Something that immediately jumps out from the results is that several top units are from the 2018 season.  This could happen for a variety of reasons.  Perhaps teams have decided to invest more in quality punt returners, or invest less in punt coverage units over the last few years.  It is also worth noting that a low number of total returns in this case does not necessarily indicate a low number of punts, as punts that went out of bounds or for touchbacks were omitted from the dataset.  ","metadata":{}},{"cell_type":"markdown","source":"# Further Applications\n## Players\nNow that we have a metric for evaluating a players **YoE**, we can use it to further analyze players in a variety of ways.  Potential followup projects include:\n* Running a clustering algorithm on **PRYOE** and its features to see if we can narrow down what physical/athletic characteristics make a successful return.\n* Run this same algorithm, but partitioned by year, then build a time-series forecasting function to predict future performance of both established players and rookies/prospects.\n* Given past college tracking data, build a function that predicts NFL **PRYOE** based on CFB **PRYOE**, opening a new avenue of draft analysis.\n\n## Units\nOf course, potential applications reach further than just player analysis.  Since a punt coverage unit is made up of 11+ players, we can examine performance in several ways.  Using our **PRYOEA** metric, we could:\n* Evaluate Special Teams coordinator performance across multiple years and teams.  Teams like the Jaguars and Seahawks seem to have consistently good **PRYOEA** numbers, what do their ST coordinators do better then most?\n* Evaluate Punt Coverage Unit performance independent of coordinator.  For example, the Packers have had multiple ST coordinators, yet their scores are consistently bad.  What can we learn from their strategy and unit composition?\n* Run a clustering algorithm on unit composition.  What player archetypes do successful units have? Are there certain characteristics that are more important than others?\n* Chart **PRYOEA** numbers for players across teams and years.  Are there players that are more impactful to their coverage unit that others? And if so, what do they have in common?","metadata":{}},{"cell_type":"markdown","source":"# Conclusion\nAlthough there is much that we can learn using the **PRYOE** metric we have developed, the number is only the beginning.  Quantifying these plays unlocks a whole treasure chest of secondary applications and analysis.  Although this model is by no means perfect, it is my hope that it can inspire and inform future player and strategy analysis and continue to advance the game.  ","metadata":{}}]}