{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Getting Started with MLB Player Digital Engagement Forecasting #","metadata":{"execution":{"iopub.status.busy":"2021-06-08T23:38:55.536027Z","iopub.execute_input":"2021-06-08T23:38:55.536369Z","iopub.status.idle":"2021-06-08T23:38:55.540097Z","shell.execute_reply.started":"2021-06-08T23:38:55.536341Z","shell.execute_reply":"2021-06-08T23:38:55.53902Z"}}},{"cell_type":"markdown","source":"Welcome to the Getting Started notebook for the *MLB Digital Engagement Forecasting* competition! This notebook is a complete start-to-finish guide to entering this competition. We will:\n- load and join the data,\n- explore time series properties,\n- create a feature set,\n- train a neural network, and\n- make a submission.\n\nAs this is a *code competition*, be aware that your submission will be a notebook to make predictions during the test period. Read more about [code competitions](https://www.kaggle.com/docs/competitions#notebooks-only-competitions).\n\nIn the complementary notebook, [Vertex AI with MLB Player Digital Engagement](https://www.kaggle.com/ryanholbrook/vertex-ai-with-mlb-player-digital-engagement), we'll also demonstrate some of the capabilities of Vertex AI, Google's new unified AI platform:\n- using Vertex AI Notebooks on Google Cloud Platform\n- exploring explainable AI on Vertex AI to refine your features","metadata":{}},{"cell_type":"code","source":"import gc\nimport sys\nimport warnings\nfrom pathlib import Path\n\nimport ipywidgets as widgets\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nfrom sklearn.model_selection import train_test_split\nfrom statsmodels.tsa.deterministic import (CalendarFourier,\n                                           CalendarSeasonality,\n                                           CalendarTimeTrend,\n                                           DeterministicProcess)\n\nimport tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras import layers\n\nwarnings.simplefilter(\"ignore\")\n\n# Set Matplotlib defaults\nplt.style.use(\"seaborn-whitegrid\")\nplt.rc(\"figure\", autolayout=True, figsize=(11, 5))\nplt.rc(\n    \"axes\",\n    labelweight=\"bold\",\n    labelsize=\"large\",\n    titleweight=\"bold\",\n    titlesize=14,\n    titlepad=10,\n)\nplot_params = dict(\n    color=\"0.75\",\n    style=\".-\",\n    markeredgecolor=\"0.25\",\n    markerfacecolor=\"0.25\",\n    legend=False,\n)","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-06-18T12:53:54.800573Z","iopub.execute_input":"2021-06-18T12:53:54.801007Z","iopub.status.idle":"2021-06-18T12:54:02.016712Z","shell.execute_reply.started":"2021-06-18T12:53:54.800922Z","shell.execute_reply":"2021-06-18T12:54:02.015739Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load and Join Data #","metadata":{}},{"cell_type":"markdown","source":"Let's take a look at some data.\n\nThe training data is a time-indexed collection of nested JSON fields containing information about each player. The targets are contained in the `nextDayPlayerEngagement` column, while the remaining columns contain data that may be used as features.","metadata":{}},{"cell_type":"code","source":"data_dir = Path('../input/mlb-player-digital-engagement-forecasting/')\ntraining = pd.read_csv(data_dir / 'train.csv')","metadata":{"execution":{"iopub.status.busy":"2021-06-18T12:54:02.018008Z","iopub.execute_input":"2021-06-18T12:54:02.018318Z","iopub.status.idle":"2021-06-18T12:55:13.318234Z","shell.execute_reply.started":"2021-06-18T12:54:02.01829Z","shell.execute_reply":"2021-06-18T12:55:13.316995Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"training[\"date\"] = pd.to_datetime(training[\"date\"], format=\"%Y%m%d\")","metadata":{"execution":{"iopub.status.busy":"2021-06-18T12:55:13.320846Z","iopub.execute_input":"2021-06-18T12:55:13.321295Z","iopub.status.idle":"2021-06-18T12:55:13.339225Z","shell.execute_reply.started":"2021-06-18T12:55:13.321248Z","shell.execute_reply":"2021-06-18T12:55:13.338122Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"training.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-18T12:55:13.34099Z","iopub.execute_input":"2021-06-18T12:55:13.341433Z","iopub.status.idle":"2021-06-18T12:55:13.389635Z","shell.execute_reply.started":"2021-06-18T12:55:13.341401Z","shell.execute_reply":"2021-06-18T12:55:13.388591Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(training.info())","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-06-18T12:55:13.391054Z","iopub.execute_input":"2021-06-18T12:55:13.391471Z","iopub.status.idle":"2021-06-18T12:55:13.414644Z","shell.execute_reply.started":"2021-06-18T12:55:13.391427Z","shell.execute_reply":"2021-06-18T12:55:13.413438Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are in addition a number of supplementary files. See the [data documentation](https://www.kaggle.com/c/mlb-player-digital-engagement-forecasting/data) on the competition page for more information.","metadata":{"execution":{"iopub.status.busy":"2021-06-13T18:13:59.278263Z","iopub.execute_input":"2021-06-13T18:13:59.278946Z","iopub.status.idle":"2021-06-13T18:13:59.287322Z","shell.execute_reply.started":"2021-06-13T18:13:59.278881Z","shell.execute_reply":"2021-06-13T18:13:59.285802Z"}}},{"cell_type":"code","source":"#### Look at data from each of the input dfs (data frames) read in ####\ndf_names = ['seasons', 'teams', 'players', 'awards']","metadata":{"execution":{"iopub.status.busy":"2021-06-18T12:55:13.416119Z","iopub.execute_input":"2021-06-18T12:55:13.416447Z","iopub.status.idle":"2021-06-18T12:55:13.42522Z","shell.execute_reply.started":"2021-06-18T12:55:13.416416Z","shell.execute_reply":"2021-06-18T12:55:13.424186Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for name in df_names:\n    globals()[name] = pd.read_csv(data_dir / f\"{name}.csv\")","metadata":{"execution":{"iopub.status.busy":"2021-06-18T12:55:13.426776Z","iopub.execute_input":"2021-06-18T12:55:13.427114Z","iopub.status.idle":"2021-06-18T12:55:13.499893Z","shell.execute_reply.started":"2021-06-18T12:55:13.427083Z","shell.execute_reply":"2021-06-18T12:55:13.498813Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"kaggle_data_tabs = widgets.Tab()\n# Add Output widgets for each pandas DF as tabs' children\nkaggle_data_tabs.children = list([widgets.Output() for df_name in df_names])","metadata":{"execution":{"iopub.status.busy":"2021-06-18T12:55:13.501153Z","iopub.execute_input":"2021-06-18T12:55:13.501426Z","iopub.status.idle":"2021-06-18T12:55:13.532129Z","shell.execute_reply.started":"2021-06-18T12:55:13.5014Z","shell.execute_reply":"2021-06-18T12:55:13.530898Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for index in range(0, len(df_names)):\n    # Rename tab bar titles to df names\n    kaggle_data_tabs.set_title(index, df_names[index])\n    \n    # Display corresponding table output for this tab name\n    with kaggle_data_tabs.children[index]:\n        display(eval(df_names[index]))\n\ndisplay(kaggle_data_tabs)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-06-18T12:55:13.535434Z","iopub.execute_input":"2021-06-18T12:55:13.535766Z","iopub.status.idle":"2021-06-18T12:55:13.64304Z","shell.execute_reply.started":"2021-06-18T12:55:13.535734Z","shell.execute_reply":"2021-06-18T12:55:13.641902Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pdb\n\n# Helper function to unpack json found in daily data\ndef unpack_json(json_str):\n    return np.nan if pd.isna(json_str) else pd.read_json(json_str)","metadata":{"execution":{"iopub.status.busy":"2021-06-18T12:55:13.645611Z","iopub.execute_input":"2021-06-18T12:55:13.646039Z","iopub.status.idle":"2021-06-18T12:55:13.65035Z","shell.execute_reply.started":"2021-06-18T12:55:13.645996Z","shell.execute_reply":"2021-06-18T12:55:13.649254Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#### Unnest various nested data within training (daily) data ####\ndaily_data_unnested_dfs = pd.DataFrame(data = {\n  'dfName': training.drop('date', axis = 1).columns.values.tolist()\n  })","metadata":{"execution":{"iopub.status.busy":"2021-06-18T12:55:13.65168Z","iopub.execute_input":"2021-06-18T12:55:13.65202Z","iopub.status.idle":"2021-06-18T12:55:13.669295Z","shell.execute_reply.started":"2021-06-18T12:55:13.651985Z","shell.execute_reply":"2021-06-18T12:55:13.668358Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In the next cell we'll restructure this data into a single dataframe `player_engagement_with_info` convienient for modeling.","metadata":{}},{"cell_type":"code","source":"daily_data_unnested_dfs['df'] = [pd.DataFrame() for row in \n  daily_data_unnested_dfs.iterrows()]","metadata":{"execution":{"iopub.status.busy":"2021-06-18T12:55:13.670438Z","iopub.execute_input":"2021-06-18T12:55:13.670847Z","iopub.status.idle":"2021-06-18T12:55:13.686581Z","shell.execute_reply.started":"2021-06-18T12:55:13.670817Z","shell.execute_reply":"2021-06-18T12:55:13.685759Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"daily_data_unnested_dfs","metadata":{"execution":{"iopub.status.busy":"2021-06-18T12:55:13.687611Z","iopub.execute_input":"2021-06-18T12:55:13.68805Z","iopub.status.idle":"2021-06-18T12:55:13.716151Z","shell.execute_reply.started":"2021-06-18T12:55:13.688019Z","shell.execute_reply":"2021-06-18T12:55:13.71547Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"training[\"nextDayPlayerEngagement\"]","metadata":{"execution":{"iopub.status.busy":"2021-06-18T12:55:13.717563Z","iopub.execute_input":"2021-06-18T12:55:13.717869Z","iopub.status.idle":"2021-06-18T12:55:13.741112Z","shell.execute_reply.started":"2021-06-18T12:55:13.717841Z","shell.execute_reply":"2021-06-18T12:55:13.740003Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for df_index, df_row in daily_data_unnested_dfs.iterrows():\n    nestedTableName = str(df_row['dfName'])\n    \n    date_nested_table = training[['date', nestedTableName]]\n    \n    date_nested_table = (date_nested_table[\n      ~pd.isna(date_nested_table[nestedTableName])\n      ].\n      reset_index(drop = True)\n      )\n    \n    daily_dfs_collection = []\n    for date_index, date_row in date_nested_table.iterrows():\n        daily_df = unpack_json(date_row[nestedTableName])\n        \n        daily_df['dailyDataDate'] = date_row['date']\n        \n        daily_dfs_collection = daily_dfs_collection + [daily_df]\n\n    unnested_table = pd.concat(daily_dfs_collection,\n      ignore_index = True).set_index('dailyDataDate').reset_index()\n\n    # Creates 1 pandas df per unnested df from daily data read in, with same name\n    globals()[df_row['dfName']] = unnested_table    \n    \n    daily_data_unnested_dfs['df'][df_index] = unnested_table\n\n","metadata":{"execution":{"iopub.status.busy":"2021-06-18T12:55:13.742681Z","iopub.execute_input":"2021-06-18T12:55:13.743084Z","iopub.status.idle":"2021-06-18T12:59:42.574792Z","shell.execute_reply.started":"2021-06-18T12:55:13.743041Z","shell.execute_reply":"2021-06-18T12:59:42.573846Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del training\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2021-06-18T12:59:42.576088Z","iopub.execute_input":"2021-06-18T12:59:42.57638Z","iopub.status.idle":"2021-06-18T12:59:42.750868Z","shell.execute_reply.started":"2021-06-18T12:59:42.576351Z","shell.execute_reply":"2021-06-18T12:59:42.749719Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"games.shape","metadata":{"execution":{"iopub.status.busy":"2021-06-18T12:59:42.752399Z","iopub.execute_input":"2021-06-18T12:59:42.752714Z","iopub.status.idle":"2021-06-18T12:59:42.766119Z","shell.execute_reply.started":"2021-06-18T12:59:42.752684Z","shell.execute_reply":"2021-06-18T12:59:42.764979Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"daily_data_unnested_dfs.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-18T12:59:42.76717Z","iopub.execute_input":"2021-06-18T12:59:42.767454Z","iopub.status.idle":"2021-06-18T12:59:43.233355Z","shell.execute_reply.started":"2021-06-18T12:59:42.767428Z","shell.execute_reply":"2021-06-18T12:59:43.232188Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#### Get some information on each date in daily data (using season dates of interest) ####\ndates = pd.DataFrame(data = \n  {'dailyDataDate': nextDayPlayerEngagement['dailyDataDate'].unique()})\n","metadata":{"execution":{"iopub.status.busy":"2021-06-18T12:59:43.234755Z","iopub.execute_input":"2021-06-18T12:59:43.23507Z","iopub.status.idle":"2021-06-18T12:59:43.25651Z","shell.execute_reply.started":"2021-06-18T12:59:43.235038Z","shell.execute_reply":"2021-06-18T12:59:43.255418Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dates['date'] = pd.to_datetime(dates['dailyDataDate'].astype(str))\n\ndates['year'] = dates['date'].dt.year\ndates['month'] = dates['date'].dt.month","metadata":{"execution":{"iopub.status.busy":"2021-06-18T12:59:43.25784Z","iopub.execute_input":"2021-06-18T12:59:43.258147Z","iopub.status.idle":"2021-06-18T12:59:43.277121Z","shell.execute_reply.started":"2021-06-18T12:59:43.258115Z","shell.execute_reply":"2021-06-18T12:59:43.276344Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dates_with_info = pd.merge(\n  dates,\n  seasons,\n  left_on = 'year',\n  right_on = 'seasonId'\n  )","metadata":{"execution":{"iopub.status.busy":"2021-06-18T12:59:43.278159Z","iopub.execute_input":"2021-06-18T12:59:43.278579Z","iopub.status.idle":"2021-06-18T12:59:43.294695Z","shell.execute_reply.started":"2021-06-18T12:59:43.278547Z","shell.execute_reply":"2021-06-18T12:59:43.2936Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dates_with_info['inSeason'] = (\n  dates_with_info['date'].between(\n    dates_with_info['regularSeasonStartDate'],\n    dates_with_info['postSeasonEndDate'],\n    inclusive = True\n    )\n  )\n","metadata":{"execution":{"iopub.status.busy":"2021-06-18T12:59:43.296564Z","iopub.execute_input":"2021-06-18T12:59:43.297056Z","iopub.status.idle":"2021-06-18T12:59:43.30875Z","shell.execute_reply.started":"2021-06-18T12:59:43.297006Z","shell.execute_reply":"2021-06-18T12:59:43.307599Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dates_with_info['seasonPart'] = np.select(\n  [\n    dates_with_info['date'] < dates_with_info['preSeasonStartDate'], \n    dates_with_info['date'] < dates_with_info['regularSeasonStartDate'],\n    dates_with_info['date'] <= dates_with_info['lastDate1stHalf'],\n    dates_with_info['date'] < dates_with_info['firstDate2ndHalf'],\n    dates_with_info['date'] <= dates_with_info['regularSeasonEndDate'],\n    dates_with_info['date'] < dates_with_info['postSeasonStartDate'],\n    dates_with_info['date'] <= dates_with_info['postSeasonEndDate'],\n    dates_with_info['date'] > dates_with_info['postSeasonEndDate']\n  ], \n  [\n    'Offseason',\n    'Preseason',\n    'Reg Season 1st Half',\n    'All-Star Break',\n    'Reg Season 2nd Half',\n    'Between Reg and Postseason',\n    'Postseason',\n    'Offseason'\n  ], \n  default = np.nan\n  )\n\n","metadata":{"execution":{"iopub.status.busy":"2021-06-18T12:59:43.309953Z","iopub.execute_input":"2021-06-18T12:59:43.310237Z","iopub.status.idle":"2021-06-18T12:59:43.337209Z","shell.execute_reply.started":"2021-06-18T12:59:43.31021Z","shell.execute_reply":"2021-06-18T12:59:43.336168Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#### Add some pitching stats/pieces of info to player game level stats ####\n\nplayer_game_stats = (playerBoxScores.copy().\n  # Change team Id/name to reflect these come from player game, not roster\n  rename(columns = {'teamId': 'gameTeamId', 'teamName': 'gameTeamName'})\n  )\n","metadata":{"execution":{"iopub.status.busy":"2021-06-18T12:59:43.338572Z","iopub.execute_input":"2021-06-18T12:59:43.338875Z","iopub.status.idle":"2021-06-18T12:59:43.463037Z","shell.execute_reply.started":"2021-06-18T12:59:43.338846Z","shell.execute_reply":"2021-06-18T12:59:43.461982Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Adds in field for innings pitched as fraction (better for aggregation)\nplayer_game_stats['inningsPitchedAsFrac'] = np.where(\n  pd.isna(player_game_stats['inningsPitched']),\n  np.nan,\n  np.floor(player_game_stats['inningsPitched']) +\n    (player_game_stats['inningsPitched'] -\n      np.floor(player_game_stats['inningsPitched'])) * 10/3\n  )\n","metadata":{"execution":{"iopub.status.busy":"2021-06-18T12:59:43.464435Z","iopub.execute_input":"2021-06-18T12:59:43.464775Z","iopub.status.idle":"2021-06-18T12:59:43.48006Z","shell.execute_reply.started":"2021-06-18T12:59:43.46474Z","shell.execute_reply":"2021-06-18T12:59:43.479086Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Add in Tom Tango pitching game score (https://www.mlb.com/glossary/advanced-stats/game-score)\nplayer_game_stats['pitchingGameScore'] = (40\n#     + 2 * player_game_stats['outs']\n    + 1 * player_game_stats['strikeOutsPitching']\n    - 2 * player_game_stats['baseOnBallsPitching']\n    - 2 * player_game_stats['hitsPitching']\n    - 3 * player_game_stats['runsPitching']\n    - 6 * player_game_stats['homeRunsPitching']\n    )","metadata":{"execution":{"iopub.status.busy":"2021-06-18T12:59:43.481138Z","iopub.execute_input":"2021-06-18T12:59:43.481583Z","iopub.status.idle":"2021-06-18T12:59:43.496053Z","shell.execute_reply.started":"2021-06-18T12:59:43.481537Z","shell.execute_reply":"2021-06-18T12:59:43.494942Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Add in criteria for no-hitter by pitcher (individual, not multiple pitchers)\nplayer_game_stats['noHitter'] = np.where(\n  (player_game_stats['gamesStartedPitching'] == 1) &\n  (player_game_stats['inningsPitched'] >= 9) &\n  (player_game_stats['hitsPitching'] == 0),\n  1, 0\n  )","metadata":{"execution":{"iopub.status.busy":"2021-06-18T12:59:43.499774Z","iopub.execute_input":"2021-06-18T12:59:43.500139Z","iopub.status.idle":"2021-06-18T12:59:43.511849Z","shell.execute_reply.started":"2021-06-18T12:59:43.500101Z","shell.execute_reply":"2021-06-18T12:59:43.510738Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"player_date_stats_agg = pd.merge(\n  (player_game_stats.\n    groupby(['dailyDataDate', 'playerId'], as_index = False).\n    # Some aggregations that are not simple sums\n    agg(\n      numGames = ('gamePk', 'nunique'),\n      # Should be 1 team per player per day, but adding here for 1 exception:\n      # playerId 518617 (Jake Diekman) had 2 games for different teams marked\n      # as played on 5/19/19, due to resumption of game after he was traded\n      numTeams = ('gameTeamId', 'nunique'),\n      # Should be only 1 team for almost all player-dates, taking min to simplify\n      gameTeamId = ('gameTeamId', 'min')\n      )\n    ),\n  # Merge with a bunch of player stats that can be summed at date/player level\n  (player_game_stats.\n    groupby(['dailyDataDate', 'playerId'], as_index = False)\n    [['runsScored', 'homeRuns', 'strikeOuts', 'baseOnBalls', 'hits',\n      'hitByPitch', 'atBats', 'caughtStealing', 'stolenBases',\n      'groundIntoDoublePlay', 'groundIntoTriplePlay', 'plateAppearances',\n      'totalBases', 'rbi', 'leftOnBase', 'sacBunts', 'sacFlies',\n      'gamesStartedPitching', 'runsPitching', 'homeRunsPitching', \n      'strikeOutsPitching', 'baseOnBallsPitching', 'hitsPitching',\n      'inningsPitchedAsFrac', 'earnedRuns', \n      'battersFaced','saves', 'blownSaves', 'pitchingGameScore', \n      'noHitter'\n      ]].\n    sum()\n    ),\n  on = ['dailyDataDate', 'playerId'],\n  how = 'inner'\n  )\n\n#### Turn games table into 1 row per team-game, then merge with team box scores ####\n# Filter to regular or Postseason games w/ valid scores for this part\ngames_for_stats = games[\n  np.isin(games['gameType'], ['R', 'F', 'D', 'L', 'W', 'C', 'P']) &\n  ~pd.isna(games['homeScore']) &\n  ~pd.isna(games['awayScore'])\n  ]\n\n# Get games table from home team perspective\ngames_home_perspective = games_for_stats.copy()\n\n# Change column names so that \"team\" is \"home\", \"opp\" is \"away\"\ngames_home_perspective.columns = [\n  col_value.replace('home', 'team').replace('away', 'opp') for \n    col_value in games_home_perspective.columns.values]\n\ngames_home_perspective['isHomeTeam'] = 1\n\n# Get games table from away team perspective\ngames_away_perspective = games_for_stats.copy()\n\n# Change column names so that \"opp\" is \"home\", \"team\" is \"away\"\ngames_away_perspective.columns = [\n  col_value.replace('home', 'opp').replace('away', 'team') for \n    col_value in games_away_perspective.columns.values]\n\ngames_away_perspective['isHomeTeam'] = 0\n\n# Put together games from home/away perspective to get df w/ 1 row per team game\nteam_games = (pd.concat([\n  games_home_perspective,\n  games_away_perspective\n  ],\n  ignore_index = True)\n  )\n\n# Copy over team box scores data to modify\nteam_game_stats = teamBoxScores.copy()\n\n# Change column names to reflect these are all \"team\" stats - helps \n# to differentiate from individual player stats if/when joining later\nteam_game_stats.columns = [\n  (col_value + 'Team') \n  if (col_value not in ['dailyDataDate', 'home', 'teamId', 'gamePk',\n    'gameDate', 'gameTimeUTC'])\n    else col_value\n  for col_value in team_game_stats.columns.values\n  ]\n\n# Merge games table with team game stats\nteam_games_with_stats = pd.merge(\n  team_games,\n  team_game_stats.\n    # Drop some fields that are already present in team_games table\n    drop(['home', 'gameDate', 'gameTimeUTC'], axis = 1),\n  on = ['dailyDataDate', 'gamePk', 'teamId'],\n  # Doing this as 'inner' join excludes spring training games, postponed games,\n  # etc. from original games table, but this may be fine for purposes here \n  how = 'inner'\n  )\n\nteam_date_stats_agg = (team_games_with_stats.\n  groupby(['dailyDataDate', 'teamId', 'gameType', 'oppId', 'oppName'], \n    as_index = False).\n  agg(\n    numGamesTeam = ('gamePk', 'nunique'),\n    winsTeam = ('teamWinner', 'sum'),\n    lossesTeam = ('oppWinner', 'sum'),\n    runsScoredTeam = ('teamScore', 'sum'),\n    runsAllowedTeam = ('oppScore', 'sum')\n    )\n   )\n\n# Prepare standings table for merge w/ player digital engagement data\n# Pick only certain fields of interest from standings for merge\nstandings_selected_fields = (standings[['dailyDataDate', 'teamId', \n  'streakCode', 'divisionRank', 'leagueRank', 'wildCardRank', 'pct'\n  ]].\n  rename(columns = {'pct': 'winPct'})\n  )\n\n# Change column names to reflect these are all \"team\" standings - helps \n# to differentiate from player-related fields if/when joining later\nstandings_selected_fields.columns = [\n  (col_value + 'Team') \n  if (col_value not in ['dailyDataDate', 'teamId'])\n    else col_value\n  for col_value in standings_selected_fields.columns.values\n  ]\n\nstandings_selected_fields['streakLengthTeam'] = (\n  standings_selected_fields['streakCodeTeam'].\n    str.replace('W', '').\n    str.replace('L', '').\n    astype(float)\n    )\n\n# Add fields to separate winning and losing streak from streak code\nstandings_selected_fields['winStreakTeam'] = np.where(\n  standings_selected_fields['streakCodeTeam'].str[0] == 'W',\n  standings_selected_fields['streakLengthTeam'],\n  np.nan\n  )\n\nstandings_selected_fields['lossStreakTeam'] = np.where(\n  standings_selected_fields['streakCodeTeam'].str[0] == 'L',\n  standings_selected_fields['streakLengthTeam'],\n  np.nan\n  )\n\nstandings_for_digital_engagement_merge = (pd.merge(\n  standings_selected_fields,\n  dates_with_info[['dailyDataDate', 'inSeason']],\n  on = ['dailyDataDate'],\n  how = 'left'\n  ).\n  # Limit down standings to only in season version\n  query(\"inSeason\").\n  # Drop fields no longer necessary (in derived values, etc.)\n  drop(['streakCodeTeam', 'streakLengthTeam', 'inSeason'], axis = 1).\n  reset_index(drop = True)\n  )\n\n#### Merge together various data frames to add date, player, roster, and team info ####\n# Copy over player engagement df to add various pieces to it\nplayer_engagement_with_info = nextDayPlayerEngagement.copy()\n\n# Take \"row mean\" across targets to add (helps with studying all 4 targets at once)\nplayer_engagement_with_info['targetAvg'] = np.mean(\n  player_engagement_with_info[['target1', 'target2', 'target3', 'target4']],\n  axis = 1)\n\n# Merge in date information\nplayer_engagement_with_info = pd.merge(\n  player_engagement_with_info,\n  dates_with_info[['dailyDataDate', 'date', 'year', 'month', 'inSeason',\n    'seasonPart']],\n  on = ['dailyDataDate'],\n  how = 'left'\n  )\n\n# Merge in some player information\nplayer_engagement_with_info = pd.merge(\n  player_engagement_with_info,\n  players[['playerId', 'playerName', 'DOB', 'mlbDebutDate', 'birthCity',\n    'birthStateProvince', 'birthCountry', 'primaryPositionName']],\n   on = ['playerId'],\n   how = 'left'\n   )\n\n# Merge in some player roster information by date\nplayer_engagement_with_info = pd.merge(\n  player_engagement_with_info,\n  (rosters[['dailyDataDate', 'playerId', 'statusCode', 'status', 'teamId']].\n    rename(columns = {\n      'statusCode': 'rosterStatusCode',\n      'status': 'rosterStatus',\n      'teamId': 'rosterTeamId'\n      })\n    ),\n  on = ['dailyDataDate', 'playerId'],\n  how = 'left'\n  )\n    \n# Merge in team name from player's roster team\nplayer_engagement_with_info = pd.merge(\n  player_engagement_with_info,\n  (teams[['id', 'teamName']].\n    rename(columns = {\n      'id': 'rosterTeamId',\n      'teamName': 'rosterTeamName'\n      })\n    ),\n  on = ['rosterTeamId'],\n  how = 'left'\n  )\n\n# Merge in some player game stats (previously aggregated) from that date\nplayer_engagement_with_info = pd.merge(\n  player_engagement_with_info,\n  player_date_stats_agg,\n  on = ['dailyDataDate', 'playerId'],\n  how = 'left'\n  )\n\n# Merge in team name from player's game team\nplayer_engagement_with_info = pd.merge(\n  player_engagement_with_info,\n  (teams[['id', 'teamName']].\n    rename(columns = {\n      'id': 'gameTeamId',\n      'teamName': 'gameTeamName'\n      })\n    ),\n  on = ['gameTeamId'],\n  how = 'left'\n  )\n\n# Merge in some team game stats/results (previously aggregated) from that date\nplayer_engagement_with_info = pd.merge(\n  player_engagement_with_info,\n  team_date_stats_agg.rename(columns = {'teamId': 'gameTeamId'}),\n  on = ['dailyDataDate', 'gameTeamId'],\n  how = 'left'\n  )\n\n# Merge in player transactions of note on that date\n    \n# Merge in some pieces of team standings (previously filter/processed) from that date\nplayer_engagement_with_info = pd.merge(\n  player_engagement_with_info,\n  standings_for_digital_engagement_merge.\n    rename(columns = {'teamId': 'gameTeamId'}),\n  on = ['dailyDataDate', 'gameTeamId'],\n  how = 'left'\n  )\n\ndisplay(player_engagement_with_info)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-06-18T12:59:43.51377Z","iopub.execute_input":"2021-06-18T12:59:43.514094Z","iopub.status.idle":"2021-06-18T12:59:57.593056Z","shell.execute_reply.started":"2021-06-18T12:59:43.514061Z","shell.execute_reply":"2021-06-18T12:59:57.592059Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Exploration #","metadata":{}},{"cell_type":"markdown","source":"Now let's do a little bit of data visualization to motivate our feature engineering. We'll investigate seasonal effects for the *Yankees* team, the team with the most overall engagement.","metadata":{}},{"cell_type":"code","source":"def seasonal_plot(X, y, period, freq, ax=None):\n    if ax is None:\n        _, ax = plt.subplots()\n    palette = sns.color_palette(\n        \"husl\",\n        n_colors=X[period].nunique(),\n    )\n    ax = sns.lineplot(\n        x=freq,\n        y=y,\n        hue=period,\n        data=X,\n        ci=False,\n        ax=ax,\n        palette=palette,\n        legend=False,\n    )\n    ax.set_title(f\"Seasonal Plot ({period}/{freq})\")\n    for line, name in zip(ax.lines, X[period].unique()):\n        y_ = line.get_ydata()[-1]\n        ax.annotate(\n            name,\n            xy=(1, y_),\n            xytext=(6, 0),\n            color=line.get_color(),\n            xycoords=ax.get_yaxis_transform(),\n            textcoords=\"offset points\",\n            size=14,\n            va=\"center\",\n        )\n    return ax\n\n\ndef plot_periodogram(ts, detrend='linear', ax=None):\n    from scipy.signal import periodogram\n    fs = pd.Timedelta(\"1Y\") / pd.Timedelta(\"1D\")\n    freqencies, spectrum = periodogram(\n        ts,\n        fs=fs,\n        detrend=detrend,\n        window=\"boxcar\",\n        scaling='spectrum',\n    )\n    if ax is None:\n        _, ax = plt.subplots()\n    ax.step(freqencies, spectrum, color=\"purple\")\n    ax.set_xscale(\"log\")\n    ax.set_xticks([1, 2, 4, 6, 12, 26, 52, 104])\n    ax.set_xticklabels(\n        [\n            \"Annual\",\n            \"Semiannual\",\n            \"Quarterly\",\n            \"Bimonthly\",\n            \"Monthly\",\n            \"Biweekly\",\n            \"Weekly\",\n            \"Semiweekly\",\n        ],\n        rotation=30,\n    )\n    ax.ticklabel_format(axis=\"y\", style=\"sci\", scilimits=(0, 0))\n    ax.set_ylabel(\"Density\")\n    ax.set_title(\"Periodogram\")\n    return ax","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-06-18T12:59:57.595259Z","iopub.execute_input":"2021-06-18T12:59:57.595723Z","iopub.status.idle":"2021-06-18T12:59:57.610576Z","shell.execute_reply.started":"2021-06-18T12:59:57.595676Z","shell.execute_reply":"2021-06-18T12:59:57.609199Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"A *seasonal plot* can reveal seasonal effects in a time series. Here are two seasonal plots for the average player engagement for the Yankees, one for yearly seasonality and one for weekly.","metadata":{}},{"cell_type":"code","source":"# Extract player engagement time series \nindex = [\"playerId\", \"date\"]\ntargets = [\"target1\", \"target2\", \"target3\", \"target4\"]\nY = player_engagement_with_info.loc[:, index + targets]\nY = Y.assign(date=lambda x: pd.to_datetime(x.date))\nY = Y.set_index(\"date\").to_period(\"D\")\nY = Y.pivot(columns=\"playerId\")\n\n# Select Yankees players\nyankees_players = player_engagement_with_info.query(\"rosterTeamName == 'Yankees'\").playerId.unique()\nyankees = Y.loc(axis=1)[:, yankees_players]\n\n# Create average engagement series\nteam = (yankees / yankees.max(axis=0)).mean(axis=1)\nteam.name = \"target\"\n\n# Yearly plot\nS = team.to_frame()\nS[\"month\"] = S.index.month  # the frequency\nS[\"year\"] = S.index.year  # the period\n_ = seasonal_plot(S, y=\"target\", period=\"year\", freq=\"month\")\n\n# Weekly plot\nS = team.to_frame()\nS[\"day\"] = S.index.dayofweek  # the frequency\nS[\"week\"] = S.index.week  # the period\n_ = seasonal_plot(S, y=\"target\", period=\"week\", freq=\"day\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-06-18T12:59:57.612143Z","iopub.execute_input":"2021-06-18T12:59:57.61256Z","iopub.status.idle":"2021-06-18T13:00:15.356216Z","shell.execute_reply.started":"2021-06-18T12:59:57.612525Z","shell.execute_reply":"2021-06-18T13:00:15.355443Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can verify this with the *periodogram*. The periodogram illustrates the strength of the frequencies within a signal -- specifically, the variance of the sine / cosine Fourier component oscillating at that frequency.","metadata":{}},{"cell_type":"code","source":"_ = plot_periodogram(team)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-06-18T13:00:15.357181Z","iopub.execute_input":"2021-06-18T13:00:15.357595Z","iopub.status.idle":"2021-06-18T13:00:15.87268Z","shell.execute_reply.started":"2021-06-18T13:00:15.357564Z","shell.execute_reply":"2021-06-18T13:00:15.871632Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Both of these visualizations indicate yearly (or annual) seasonality in the engagement time series. We'll model this effect with Fourier features, which we create below.","metadata":{}},{"cell_type":"markdown","source":"# Data Pipeline #","metadata":{}},{"cell_type":"markdown","source":"The following cell creates our data splits for training. We will:\n- select a set of features from `player_engagement_with_info`\n- pivot the dataframe from long to wide format so that each column comprises a time series for the training period\n- create Fourier features to model the annual seasonality\n- split data into training and validation sets","metadata":{}},{"cell_type":"code","source":"df = player_engagement_with_info\n\n# Columns to select\nindex = [\"playerId\", \"date\"]\ntargets = [\"target1\", \"target2\", \"target3\", \"target4\"]\nfeatures = [\n    \"runsScored\", \"hits\", \"stolenBases\", \"rbi\", \"atBats\", \"saves\", \"homeRuns\",\n    \"stolenBases\", \"strikeOuts\",\n]\n\n# Targets\nY = df.loc[:, index + targets]\nY = Y.assign(date=lambda x: pd.to_datetime(x.date))\nY = Y.set_index(\"date\").to_period(\"D\")\nY = Y.pivot(columns=\"playerId\")\n\n# Features\nX = df.loc[:, index + features].set_index(\"date\").to_period(\"D\")\nfor col in features:\n    X[col] = X[col].fillna(-1)\nX = X.pivot(columns=[\"playerId\"])\n\n# Temporal features\nfourier = CalendarFourier(freq='A', order=8)\ndeterministic = DeterministicProcess(\n    index=X.index,\n    order=0,\n    seasonal=False,  # set to True to create indicators for days of the week (weekly seasonality)\n    additional_terms=[fourier],\n)\nX = pd.concat([X, deterministic.in_sample()], axis=1)\n\n# Training splits\nX_train, X_valid, y_train, y_valid = train_test_split(\n    X,\n    Y,\n    test_size=30,  # 30-day validation set\n    shuffle=False,\n)","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2021-06-18T13:00:15.874038Z","iopub.execute_input":"2021-06-18T13:00:15.874356Z","iopub.status.idle":"2021-06-18T13:00:19.08182Z","shell.execute_reply.started":"2021-06-18T13:00:15.874317Z","shell.execute_reply":"2021-06-18T13:00:19.080759Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model #","metadata":{}},{"cell_type":"markdown","source":"For this getting started notebook, we'll just create a simple feedforward network.","metadata":{}},{"cell_type":"code","source":"OUTPUTS = y_train.shape[-1]\nearly_stopping = keras.callbacks.EarlyStopping(patience=5,\n                                               restore_best_weights=True)\n\nmodel = keras.Sequential([\n    layers.Dense(1024, activation='relu'),\n    layers.BatchNormalization(),\n    layers.Dropout(0.2),\n    layers.Dense(1024, activation='relu'),\n    layers.BatchNormalization(),\n    layers.Dropout(0.2),\n    layers.Dense(1024, activation='relu'),\n    layers.BatchNormalization(),\n    layers.Dropout(0.2),\n    layers.Dense(OUTPUTS),\n])\n","metadata":{"execution":{"iopub.status.busy":"2021-06-18T13:00:19.083262Z","iopub.execute_input":"2021-06-18T13:00:19.083873Z","iopub.status.idle":"2021-06-18T13:00:19.163419Z","shell.execute_reply.started":"2021-06-18T13:00:19.083827Z","shell.execute_reply":"2021-06-18T13:00:19.16252Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.compile(\n    optimizer='adam',\n    loss='mae',\n    metrics=['mae'],\n)\nhistory = model.fit(\n    X_train,\n    y_train,\n    validation_data=(X_valid, y_valid),\n    epochs=50,\n    callbacks=[early_stopping]\n)","metadata":{"execution":{"iopub.status.busy":"2021-06-18T13:00:19.164604Z","iopub.execute_input":"2021-06-18T13:00:19.164918Z","iopub.status.idle":"2021-06-18T13:01:24.863153Z","shell.execute_reply.started":"2021-06-18T13:00:19.164889Z","shell.execute_reply":"2021-06-18T13:01:24.861895Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Evaluate #","metadata":{}},{"cell_type":"code","source":"y_eval = model.predict(X_valid)\ny_eval = pd.DataFrame(y_eval,\n                      index=y_valid.index,\n                      columns=y_valid.columns)\ny_fit = model.predict(X_train)\ny_fit = pd.DataFrame(y_fit,\n                     index=y_train.index,\n                     columns=y_train.columns)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-06-18T13:01:24.865851Z","iopub.execute_input":"2021-06-18T13:01:24.866304Z","iopub.status.idle":"2021-06-18T13:01:26.872692Z","shell.execute_reply.started":"2021-06-18T13:01:24.866257Z","shell.execute_reply":"2021-06-18T13:01:26.871849Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's look at predictions for Aaron Judge of the NY Yankees, the player with the overall highest engagement.","metadata":{}},{"cell_type":"code","source":"player = 592450  # Aaron Judge of the NY Yankees\nfig, ax = plt.subplots(figsize=(11, 8))\nax = y_eval.loc(axis=1)[:, player].plot(ax=ax, subplots=True, color='C0')\nax = y_valid.loc(axis=1)[:, player].plot(subplots=True,\n                                         sharex=True,\n                                         ax=ax,\n                                         color='0.25')\nax = y_fit.loc(axis=1)[:, player].plot(subplots=True,\n                                       sharex=True,\n                                       ax=ax,\n                                       color='C3')\nax = y_train.loc(axis=1)[:, player].plot(subplots=True,\n                                         sharex=True,\n                                         ax=ax,\n                                         color='0.25')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-06-18T13:01:26.874289Z","iopub.execute_input":"2021-06-18T13:01:26.8749Z","iopub.status.idle":"2021-06-18T13:01:28.544405Z","shell.execute_reply.started":"2021-06-18T13:01:26.874854Z","shell.execute_reply":"2021-06-18T13:01:28.543195Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Create Submission #","metadata":{}},{"cell_type":"markdown","source":"The next cell illustrates how you to create a submission for this competition. As this is a code competition that relies on a time series module, submissions must follow the requirements described on the [Evaluation Page](https://www.kaggle.com/c/mlb-player-digital-engagement-forecasting/overview/evaluation) and [Data Page](https://www.kaggle.com/c/mlb-player-digital-engagement-forecasting/data).","metadata":{}},{"cell_type":"code","source":"if 'kaggle_secrets' in sys.modules:  # only run while on Kaggle\n    import mlb\n\n    env = mlb.make_env()\n    iter_test = env.iter_test()\n\n    for (test_df, sample_prediction_df) in iter_test:\n    \n        # Example: unpack a dataframe from a json column\n        today_games = unpack_json(test_df['games'].iloc[0])\n    \n        # Make your predictions for the next day's engagement\n        sample_prediction_df['target1'] = 100.00\n    \n        # Submit your predictions \n        env.predict(sample_prediction_df)","metadata":{"execution":{"iopub.status.busy":"2021-06-18T13:01:28.545743Z","iopub.execute_input":"2021-06-18T13:01:28.546048Z","iopub.status.idle":"2021-06-18T13:01:30.21836Z","shell.execute_reply.started":"2021-06-18T13:01:28.546018Z","shell.execute_reply":"2021-06-18T13:01:30.217614Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Explainable AI with Vertex #\n\n[This complementary notebook](https://www.kaggle.com/ryanholbrook/vertex-ai-with-mlb-player-digital-engagement) demonstrates how to run this notebook in Vertex AI Notebooks and use Explainable AI (XAI) to refine your features.","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}