{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# MLB Player Digital Engagement Forecasting","metadata":{}},{"cell_type":"markdown","source":"- In this notebook, we have used features from playersBoxScores, teamBoxScores, transactions, standings and awards from train dataset.\n- Simple NN model was used for making predictions on MLB player digital engagement scores.","metadata":{}},{"cell_type":"markdown","source":"## Data Preprocessing I : Train.csv\n\n- We have separated each features in the train set and used different approaches on each features - for example, filling missing values or changing monthly data into daily view using interpolation","metadata":{}},{"cell_type":"code","source":"# Data Preprocessing\nimport pandas as pd\nimport numpy as np\nimport json\n\nimport os\nimport time\nfrom datetime import datetime, timedelta\n\n# Modeling\nimport tensorflow as tf\nfrom tensorflow import keras\nimport matplotlib.pyplot as plt\nimport random\nfrom tensorflow.keras import models\nfrom tensorflow.keras import layers","metadata":{"execution":{"iopub.status.busy":"2021-07-31T14:55:48.757213Z","iopub.execute_input":"2021-07-31T14:55:48.757561Z","iopub.status.idle":"2021-07-31T14:55:54.342195Z","shell.execute_reply.started":"2021-07-31T14:55:48.757481Z","shell.execute_reply":"2021-07-31T14:55:54.341079Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- To reduce time for the data processing, we will leave the code how we read and processed train_updated.csv, however the entire process will be done by reading csv files which were separated before creating this notebook.","metadata":{}},{"cell_type":"code","source":"# fileDir = \"../input/mlb-player-digital-engagement-forecasting/\"\n\n# train_path = os.path.join(fileDir, \"train_updated.csv\")\n# train = pd.read_csv(train_path)","metadata":{"execution":{"iopub.status.busy":"2021-07-31T12:53:20.654836Z","iopub.execute_input":"2021-07-31T12:53:20.655227Z","iopub.status.idle":"2021-07-31T12:53:20.659979Z","shell.execute_reply.started":"2021-07-31T12:53:20.655198Z","shell.execute_reply":"2021-07-31T12:53:20.658472Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# def dict_to_df(df):\n    \n#     result = pd.DataFrame()\n    \n#     for i in range(len(df)):\n#         temp_dict = df[i]\n        \n#         # some records have NaN \n#         if type(temp_dict) == float:\n#             pass\n#         else:\n#             temp_df = pd.DataFrame.from_dict(json.loads(temp_dict))\n#             result = result.append(temp_df, ignore_index = True)\n        \n#     return(result)","metadata":{"execution":{"iopub.status.busy":"2021-07-31T14:55:57.434456Z","iopub.execute_input":"2021-07-31T14:55:57.434885Z","iopub.status.idle":"2021-07-31T14:55:57.443477Z","shell.execute_reply.started":"2021-07-31T14:55:57.434854Z","shell.execute_reply":"2021-07-31T14:55:57.441964Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### start = time.time()\n\n# rosters = dict_to_df(train[\"rosters\"])\n# games = dict_to_df(train[\"games\"])\n# playerBoxScores = dict_to_df(train[\"playerBoxScores\"])\n# teamBoxScores = dict_to_df(train[\"teamBoxScores\"])\n# transactions = dict_to_df(train[\"transactions\"])\n# standings = dict_to_df(train[\"standings\"])\n# awards = dict_to_df(train[\"awards\"])\n# events = dict_to_df(train[\"events\"])\n# playerTwitterFollowers = dict_to_df(train[\"playerTwitterFollowers\"])\n# teamTwitterFollowers = dict_to_df(train[\"teamTwitterFollowers\"])\n\n# target = dict_to_df(train[\"nextDayPlayerEngagement\"])\n\n# print(time.time() - start, \"seconds\")","metadata":{}},{"cell_type":"markdown","source":"## Data Preprocessing II : Merge csv files\n\n- For those datasets not included in train.csv, we have loaded each data separately and took another steps of data preprocessing.\n- For players.csv, we filtered playerIds which are chosen for the test set.\n- For seasons.csv, we labeled each date for different types (Preseason, Regular season, etc.) to indicate different event types.\n- For awards.csv, we mutated data to use the information of number of awards received by each players.","metadata":{}},{"cell_type":"code","source":"fileDir = \"../input/mlb-player-digital-engagement-forecasting/\"\n\ndfs = {}\nfor fileName in [\"players.csv\", \"seasons.csv\", \"teams.csv\"]:\n    key = fileName.split(\".\")[0]\n    idx = pd.read_csv(fileDir + fileName)\n    \n    dfs[key] = idx","metadata":{"execution":{"iopub.status.busy":"2021-07-31T14:56:05.626954Z","iopub.execute_input":"2021-07-31T14:56:05.627285Z","iopub.status.idle":"2021-07-31T14:56:05.668358Z","shell.execute_reply.started":"2021-07-31T14:56:05.627257Z","shell.execute_reply":"2021-07-31T14:56:05.667418Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Players","metadata":{}},{"cell_type":"code","source":"players = dfs[\"players\"]\nplayersForTest = players[players[\"playerForTestSetAndFuturePreds\"] == True]\nprint(\"Players for test set:\", len(playersForTest))","metadata":{"execution":{"iopub.status.busy":"2021-07-31T14:56:05.67025Z","iopub.execute_input":"2021-07-31T14:56:05.670821Z","iopub.status.idle":"2021-07-31T14:56:05.699195Z","shell.execute_reply.started":"2021-07-31T14:56:05.67078Z","shell.execute_reply":"2021-07-31T14:56:05.697573Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Seasons","metadata":{}},{"cell_type":"code","source":"seasons = dfs[\"seasons\"]\n\nseasons_df = pd.DataFrame()\n\nfor i in range(len(seasons)):\n    seasonId = seasons[\"seasonId\"][i]\n    yearStart = str(seasonId) + \"-01-01\"\n    yearEnd = str(seasonId+1) + \"-01-01\"\n    \n    yearDates = pd.date_range(yearStart, yearEnd).date\n    \n    df = pd.DataFrame({\"date\": yearDates, \"seasonIdx\": np.full_like(len(yearDates), np.nan, dtype = np.double())})\n\n    \n    for index in df.index:\n        df.at[index,\"date\"] = str(df.at[index,\"date\"]) \n    \n    preSeasonStartDate = seasons[\"preSeasonStartDate\"][i]\n    preSeasonEndDate = seasons[\"preSeasonEndDate\"][i]\n    regularSeasonStartDate = seasons[\"regularSeasonStartDate\"][i]\n    regularSeasonEndDate = seasons[\"regularSeasonEndDate\"][i]\n    allStarDate = seasons[\"allStarDate\"][i]\n    postSeasonStartDate = seasons[\"postSeasonStartDate\"][i]\n    postSeasonEndDate = seasons[\"postSeasonEndDate\"][i]\n    \n    df[df[\"date\"] == preSeasonStartDate] = df[df[\"date\"] == preSeasonStartDate].fillna(1)\n    df[df[\"date\"] == preSeasonEndDate] = df[df[\"date\"] == preSeasonEndDate].fillna(1)\n    df[df[\"date\"] == regularSeasonStartDate] = df[df[\"date\"] == regularSeasonStartDate].fillna(2)\n    df[df[\"date\"] == regularSeasonEndDate] = df[df[\"date\"] == regularSeasonEndDate].fillna(2)\n    df[df[\"date\"] == postSeasonStartDate] = df[df[\"date\"] == postSeasonStartDate].fillna(4)\n    df[df[\"date\"] == postSeasonEndDate] = df[df[\"date\"] == postSeasonEndDate].fillna(4)\n    \n    df = df.fillna(method = 'ffill')\n    \n    df[df[\"date\"] == allStarDate] = 3\n    \n    df = df.fillna(0)\n\n    \n    seasons_df = seasons_df.append(df, ignore_index = True)\n\nseasons_df.loc[seasons_df[\"seasonIdx\"]>0]","metadata":{"execution":{"iopub.status.busy":"2021-07-31T14:56:05.703097Z","iopub.execute_input":"2021-07-31T14:56:05.703822Z","iopub.status.idle":"2021-07-31T14:56:05.905223Z","shell.execute_reply.started":"2021-07-31T14:56:05.703777Z","shell.execute_reply":"2021-07-31T14:56:05.903867Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# deleting dfs and train to save RAM memories.\n# del dfs\n# del train\ndel seasons","metadata":{"execution":{"iopub.status.busy":"2021-07-31T14:56:05.907494Z","iopub.execute_input":"2021-07-31T14:56:05.907976Z","iopub.status.idle":"2021-07-31T14:56:05.914384Z","shell.execute_reply.started":"2021-07-31T14:56:05.907934Z","shell.execute_reply":"2021-07-31T14:56:05.912705Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Awards","metadata":{}},{"cell_type":"code","source":"def date_playerId(df, date, playerId = \"playerId\"):\n    \n    df[\"date_playerId\"] = df[date].astype(str).str.replace(\"-\", \"\") + \"_\" + df[playerId].astype(str).str.replace(\"\\.0\", \"\", regex = True)\n    \n    return df","metadata":{"execution":{"iopub.status.busy":"2021-07-31T14:56:05.916257Z","iopub.execute_input":"2021-07-31T14:56:05.916981Z","iopub.status.idle":"2021-07-31T14:56:05.926926Z","shell.execute_reply.started":"2021-07-31T14:56:05.916923Z","shell.execute_reply":"2021-07-31T14:56:05.925422Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fileDir = \"../input/mlbcsv\"\nawards = pd.read_csv(fileDir + \"/awards.csv\")\n\nawards = date_playerId(awards, \"awardDate\")[[\"date_playerId\", \"playerId\", \"awardId\"]]\nawards[\"isAwarded\"] = np.ones(len(awards))\nawards[\"cumulativeAwardScore\"] = awards.groupby(\"playerId\")[\"isAwarded\"].transform(pd.Series.cumsum)\nawards = awards.drop(columns = [\"awardId\"])\nawards.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-31T14:56:05.928805Z","iopub.execute_input":"2021-07-31T14:56:05.929718Z","iopub.status.idle":"2021-07-31T14:56:06.952684Z","shell.execute_reply.started":"2021-07-31T14:56:05.92962Z","shell.execute_reply":"2021-07-31T14:56:06.951328Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- For the rest of the features in train.csv, we have dropped unnecessary columns and created primary key date_playerId for further uses\n\n### Transactions","metadata":{}},{"cell_type":"code","source":"fileDir = \"../input/mlbcsv\"\ntransactions = pd.read_csv(fileDir + \"/transactions.csv\")\n\ntransactions = transactions.dropna(subset = [\"playerId\"]).reset_index(drop = True)\ntransactions = date_playerId(transactions, \"date\")[[\"date_playerId\", \"fromTeamId\", \"toTeamId\", \"effectiveDate\",\"typeCode\"]]\ntransactions.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-31T14:56:06.954934Z","iopub.execute_input":"2021-07-31T14:56:06.955406Z","iopub.status.idle":"2021-07-31T14:56:07.347396Z","shell.execute_reply.started":"2021-07-31T14:56:06.955363Z","shell.execute_reply":"2021-07-31T14:56:07.346449Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Standings","metadata":{}},{"cell_type":"code","source":"fileDir = \"../input/mlbcsv\"\nstandings = pd.read_csv(fileDir + \"/standings.csv\")\n\nstandings = standings[[\"season\",\"gameDate\", \"teamId\", \"streakCode\", \"divisionRank\", \"leagueRank\", \"wins\", \"losses\", \n                       \"pct\", \"divisionChamp\", \"divisionLeader\"]]\nstandings.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-31T14:56:07.351039Z","iopub.execute_input":"2021-07-31T14:56:07.351445Z","iopub.status.idle":"2021-07-31T14:56:07.499839Z","shell.execute_reply.started":"2021-07-31T14:56:07.351411Z","shell.execute_reply":"2021-07-31T14:56:07.498489Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"divisionRank_byDates = {}\n\nfor index in standings.index:\n    gamedate = standings.at[index,\"gameDate\"][:4] + standings.at[index,\"gameDate\"][5:7] + standings.at[index,\"gameDate\"][8:10]\n    teamid = standings.at[index,\"teamId\"]\n    \n    streakCode = standings.at[index,\"streakCode\"]\n    divisionRank = standings.at[index,\"divisionRank\"]\n    leagueRank = standings.at[index,\"leagueRank\"]\n    pct = standings.at[index,\"pct\"]\n    divisionChamp = standings.at[index,\"divisionChamp\"]\n    divisionLeader = standings.at[index,\"divisionLeader\"]\n    \n    \n    if teamid not in divisionRank_byDates.keys():\n        divisionRank_byDates[teamid] = {gamedate:[streakCode,divisionRank,leagueRank,pct,divisionChamp,divisionLeader]}\n\n    else:\n        divisionRank_byDates[teamid][gamedate] = [streakCode,divisionRank,leagueRank,pct,divisionChamp,divisionLeader]","metadata":{"execution":{"iopub.status.busy":"2021-07-31T14:56:07.502994Z","iopub.execute_input":"2021-07-31T14:56:07.503496Z","iopub.status.idle":"2021-07-31T14:56:08.653951Z","shell.execute_reply.started":"2021-07-31T14:56:07.503451Z","shell.execute_reply":"2021-07-31T14:56:08.65277Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Team Twitter Followers","metadata":{}},{"cell_type":"code","source":"fileDir = \"../input/mlbcsv\"\nteamTwitterFollowers = pd.read_csv(fileDir + \"/teamTwitterFollowers.csv\")\n\nteamTwitterFollowers = teamTwitterFollowers[[\"date\", \"teamId\", \"numberOfFollowers\"]]\nteamTwitterFollowers = teamTwitterFollowers.rename(columns={\"numberOfFollowers\": \"teamFollowers\"})\n\nteamTwitterFollowers[\"date\"] = pd.to_datetime(teamTwitterFollowers[\"date\"])\n\nteamTwitter = pd.DataFrame()\n\nminDate = min(teamTwitterFollowers[\"date\"])\nmaxDate = max(teamTwitterFollowers[\"date\"]) + timedelta(days = 30)\n\nfor teamId in teamTwitterFollowers[\"teamId\"].unique():\n    df = teamTwitterFollowers[teamTwitterFollowers[\"teamId\"] == teamId]\n    df = df.set_index(df[\"date\"]).drop(columns = [\"date\"])\n    \n    idx = pd.date_range(minDate, maxDate)\n    df = df.reindex(idx, fill_value = np.nan)\n    df[\"teamId\"] = df[\"teamId\"].interpolate(method = \"pad\")\n    df[\"teamFollowers\"] = df[\"teamFollowers\"].interpolate(method = \"linear\", limit_direction = \"forward\")\n\n    teamTwitter = teamTwitter.append(df)\n\ndel teamTwitterFollowers\n\nteamTwitter = teamTwitter.reset_index()\nteamTwitter = teamTwitter.rename(columns={\"index\": \"date\"})\n\n# converting timestamp to string (up to YYYY-MM-DD)\nteamTwitter[\"date\"] = teamTwitter[\"date\"].astype(str)\n\nteamTwitter.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-31T14:56:08.655547Z","iopub.execute_input":"2021-07-31T14:56:08.656307Z","iopub.status.idle":"2021-07-31T14:56:09.099649Z","shell.execute_reply.started":"2021-07-31T14:56:08.656227Z","shell.execute_reply":"2021-07-31T14:56:09.098538Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Player Twitter Followers","metadata":{}},{"cell_type":"code","source":"fileDir = \"../input/mlbcsv\"\nplayerTwitterFollowers = pd.read_csv(fileDir + \"/playerTwitterFollowers.csv\")\n\nplayerTwitterFollowers = playerTwitterFollowers[[\"date\", \"playerId\", \"numberOfFollowers\"]]\nplayerTwitterFollowers[\"date\"] = pd.to_datetime(playerTwitterFollowers[\"date\"])\n\nplayerTwitter = pd.DataFrame()\n\nminDate = min(playerTwitterFollowers[\"date\"])\nmaxDate = max(playerTwitterFollowers[\"date\"]) + timedelta(days = 30)\n\nfor playerId in playerTwitterFollowers[\"playerId\"].unique():\n    df = playerTwitterFollowers[playerTwitterFollowers[\"playerId\"] == playerId]\n    df = df.set_index(df[\"date\"]).drop(columns = [\"date\"])\n    \n    idx = pd.date_range(minDate, maxDate)\n    df = df.reindex(idx, fill_value = np.nan)\n    df[\"playerId\"] = df[\"playerId\"].interpolate(method = \"pad\")\n    df[\"numberOfFollowers\"] = df[\"numberOfFollowers\"].interpolate(method = \"linear\", limit_direction = \"forward\")\n\n    playerTwitter = playerTwitter.append(df)\n\ndel playerTwitterFollowers    \n\nplayerTwitter.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-31T14:56:09.101392Z","iopub.execute_input":"2021-07-31T14:56:09.101861Z","iopub.status.idle":"2021-07-31T14:56:24.99389Z","shell.execute_reply.started":"2021-07-31T14:56:09.10182Z","shell.execute_reply":"2021-07-31T14:56:24.992866Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Rosters","metadata":{}},{"cell_type":"code","source":"fileDir = \"../input/mlbcsv\"\nrosters = pd.read_csv(fileDir + \"/rosters.csv\").iloc[:, 1:]\n\nrosters = date_playerId(rosters, \"gameDate\").drop(columns = [\"status\", \"gameDate\"])\nrosters.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-31T14:56:24.995335Z","iopub.execute_input":"2021-07-31T14:56:24.995844Z","iopub.status.idle":"2021-07-31T14:56:31.71464Z","shell.execute_reply.started":"2021-07-31T14:56:24.9958Z","shell.execute_reply":"2021-07-31T14:56:31.713512Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Box Scores\n\n- For features in player box scores, we have separated features in to 3 different types: batting, pitching and defense.\n- For each features, we have applied PCA to reduce the dimensions which will be used in later modeling.","metadata":{}},{"cell_type":"code","source":"fileDir = \"../input/mlbcsv\"\nplayerBoxScores = pd.read_csv(fileDir + \"/playerBoxScores.csv\").iloc[:, 1:]\n\nplayerBoxScores.columns","metadata":{"execution":{"iopub.status.busy":"2021-07-31T14:56:31.7164Z","iopub.execute_input":"2021-07-31T14:56:31.716903Z","iopub.status.idle":"2021-07-31T14:56:33.54496Z","shell.execute_reply.started":"2021-07-31T14:56:31.716859Z","shell.execute_reply":"2021-07-31T14:56:33.543954Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"battingStats = ['flyOuts', 'groundOuts', 'runsScored', 'doubles', 'triples', 'homeRuns',\n                'strikeOuts', 'baseOnBalls', 'intentionalWalks', 'hits', 'hitByPitch',\n                'atBats', 'caughtStealing', 'stolenBases', 'groundIntoDoublePlay',\n                'groundIntoTriplePlay', 'plateAppearances', 'totalBases', 'rbi',\n                'leftOnBase', 'sacBunts', 'sacFlies', 'catchersInterference',\n                'pickoffs']\n\npitchingStats = ['flyOutsPitching', 'airOutsPitching',\n                 'groundOutsPitching', 'runsPitching', 'doublesPitching',\n                 'triplesPitching', 'homeRunsPitching', 'strikeOutsPitching',\n                 'baseOnBallsPitching', 'intentionalWalksPitching', 'hitsPitching',\n                 'hitByPitchPitching', 'atBatsPitching', 'caughtStealingPitching',\n                 'stolenBasesPitching', 'inningsPitched', 'saveOpportunities',\n                 'earnedRuns', 'battersFaced', 'outsPitching', 'pitchesThrown', 'balls',\n                 'strikes', 'hitBatsmen', 'balks', 'wildPitches', 'pickoffsPitching',\n                 'rbiPitching', 'gamesFinishedPitching', 'inheritedRunners',\n                 'inheritedRunnersScored', 'catchersInterferencePitching',\n                 'sacBuntsPitching', 'sacFliesPitching', 'saves', 'holds', 'blownSaves']\n\ndefStats = ['assists', 'putOuts', 'errors', 'chances']","metadata":{"execution":{"iopub.status.busy":"2021-07-31T14:56:33.548279Z","iopub.execute_input":"2021-07-31T14:56:33.548597Z","iopub.status.idle":"2021-07-31T14:56:33.556904Z","shell.execute_reply.started":"2021-07-31T14:56:33.548552Z","shell.execute_reply":"2021-07-31T14:56:33.555655Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def high_corr_cols(df, threshold = 0.7):\n    for col in range(len(df)):\n        for row in range(col+1, len(df)):\n            if 0.7 <= df.iloc[row, col] < 1:\n                print(df.index[row], df.columns[col], df.iloc[row, col])","metadata":{"execution":{"iopub.status.busy":"2021-07-31T14:56:33.558506Z","iopub.execute_input":"2021-07-31T14:56:33.559197Z","iopub.status.idle":"2021-07-31T14:56:33.572149Z","shell.execute_reply.started":"2021-07-31T14:56:33.559153Z","shell.execute_reply":"2021-07-31T14:56:33.570975Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"battingCorr = playerBoxScores[battingStats].corr().abs()\nhigh_corr_cols(battingCorr)\n\n# drop totalBases, plateAppearances","metadata":{"execution":{"iopub.status.busy":"2021-07-31T14:56:33.574022Z","iopub.execute_input":"2021-07-31T14:56:33.574575Z","iopub.status.idle":"2021-07-31T14:56:33.876369Z","shell.execute_reply.started":"2021-07-31T14:56:33.574523Z","shell.execute_reply":"2021-07-31T14:56:33.875037Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pitchingCorr = playerBoxScores[pitchingStats].corr().abs()\nhigh_corr_cols(pitchingCorr)\n\n# drop atBatsPitching, airOutsPitching, pitchesThrown, inningsPitched, outsPitching, battersFaced","metadata":{"execution":{"iopub.status.busy":"2021-07-31T14:56:33.878009Z","iopub.execute_input":"2021-07-31T14:56:33.878673Z","iopub.status.idle":"2021-07-31T14:56:34.319491Z","shell.execute_reply.started":"2021-07-31T14:56:33.878618Z","shell.execute_reply":"2021-07-31T14:56:34.318439Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"defCorr = playerBoxScores[defStats].corr().abs()\nhigh_corr_cols(defCorr)\n\n# drop chances","metadata":{"execution":{"iopub.status.busy":"2021-07-31T14:56:34.32196Z","iopub.execute_input":"2021-07-31T14:56:34.322434Z","iopub.status.idle":"2021-07-31T14:56:34.348098Z","shell.execute_reply.started":"2021-07-31T14:56:34.322365Z","shell.execute_reply":"2021-07-31T14:56:34.347152Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"playerBoxScores = playerBoxScores.drop(columns = [\"gameTimeUTC\", \"teamName\", \"playerName\", \"jerseyNum\", \"positionName\", \"positionType\",\n                                                  \"totalBases\", \"plateAppearances\", \"atBatsPitching\", \"airOutsPitching\", \"pitchesThrown\",\n                                                  \"inningsPitched\", \"outsPitching\", \"battersFaced\", \"chances\"])\nplayerBoxScores.tail()","metadata":{"execution":{"iopub.status.busy":"2021-07-31T14:56:34.349811Z","iopub.execute_input":"2021-07-31T14:56:34.35025Z","iopub.status.idle":"2021-07-31T14:56:34.440007Z","shell.execute_reply.started":"2021-07-31T14:56:34.350221Z","shell.execute_reply":"2021-07-31T14:56:34.438596Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"playerBoxScores = date_playerId(playerBoxScores, \"gameDate\")\nplayerBoxScores.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-31T14:56:34.441801Z","iopub.execute_input":"2021-07-31T14:56:34.442216Z","iopub.status.idle":"2021-07-31T14:56:35.047151Z","shell.execute_reply.started":"2021-07-31T14:56:34.442176Z","shell.execute_reply":"2021-07-31T14:56:35.045834Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"playerBoxScores.columns","metadata":{"execution":{"iopub.status.busy":"2021-07-31T14:56:35.048869Z","iopub.execute_input":"2021-07-31T14:56:35.049316Z","iopub.status.idle":"2021-07-31T14:56:35.058657Z","shell.execute_reply.started":"2021-07-31T14:56:35.049271Z","shell.execute_reply":"2021-07-31T14:56:35.057292Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"battingFeat = ['flyOuts', 'groundOuts', 'runsScored', 'doubles', 'triples', 'homeRuns', 'strikeOuts',\n               'baseOnBalls', 'intentionalWalks', 'hits', 'hitByPitch', 'atBats', 'caughtStealing', 'stolenBases', 'groundIntoDoublePlay',\n               'groundIntoTriplePlay', 'rbi', 'leftOnBase', 'sacBunts', 'sacFlies', 'catchersInterference', 'pickoffs']\n\nbatPCA = playerBoxScores[battingFeat].fillna(0)\n\nfrom sklearn.decomposition import PCA\nimport matplotlib.pyplot as plt\n\nn_comp, var_ratio = [], []\n\nfor i in range(1, 21):\n    pca = PCA(n_components = i, random_state = 1)\n    pca.fit(batPCA)\n    \n    n_comp.append(i)\n    var_ratio.append(pca.explained_variance_ratio_.sum())\n    \nplt.plot(n_comp, var_ratio)\nplt.show()\n\n# n = 7","metadata":{"execution":{"iopub.status.busy":"2021-07-31T14:56:35.060685Z","iopub.execute_input":"2021-07-31T14:56:35.061727Z","iopub.status.idle":"2021-07-31T14:56:51.932594Z","shell.execute_reply.started":"2021-07-31T14:56:35.061683Z","shell.execute_reply":"2021-07-31T14:56:51.931639Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"bat_pca = PCA(n_components = 7, random_state = 1)\nbat_pca.fit(playerBoxScores[battingFeat].fillna(0))","metadata":{"execution":{"iopub.status.busy":"2021-07-31T14:56:51.938982Z","iopub.execute_input":"2021-07-31T14:56:51.939303Z","iopub.status.idle":"2021-07-31T14:56:52.708158Z","shell.execute_reply.started":"2021-07-31T14:56:51.939273Z","shell.execute_reply":"2021-07-31T14:56:52.706877Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"batPCAdf = pd.concat([playerBoxScores[playerBoxScores[\"gamesPlayedBatting\"] == 1][\"date_playerId\"], playerBoxScores[playerBoxScores[\"gamesPlayedBatting\"] == 1][battingFeat]], axis = 1)\nbatPCAdf = batPCAdf.fillna(0).reset_index(drop = True)","metadata":{"execution":{"iopub.status.busy":"2021-07-31T14:56:52.713592Z","iopub.execute_input":"2021-07-31T14:56:52.717634Z","iopub.status.idle":"2021-07-31T14:56:52.896626Z","shell.execute_reply.started":"2021-07-31T14:56:52.717565Z","shell.execute_reply":"2021-07-31T14:56:52.895472Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"bat_df = pd.concat([batPCAdf.iloc[:, 0], pd.DataFrame(bat_pca.fit_transform(batPCAdf.iloc[:, 1:]))], axis = 1)\nbat_df = bat_df.rename(columns = {0: \"bat0\", 1: \"bat1\", 2: \"bat2\", 3: \"bat3\", 4: \"bat4\", 5: \"bat5\", 6: \"bat6\"})\nbat_df","metadata":{"execution":{"iopub.status.busy":"2021-07-31T14:56:52.898281Z","iopub.execute_input":"2021-07-31T14:56:52.898722Z","iopub.status.idle":"2021-07-31T14:56:53.503467Z","shell.execute_reply.started":"2021-07-31T14:56:52.898677Z","shell.execute_reply":"2021-07-31T14:56:53.501977Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pitchingFeat = ['gamesStartedPitching', 'completeGamesPitching', 'shutoutsPitching', 'winsPitching', 'lossesPitching', 'flyOutsPitching',\n                'groundOutsPitching', 'runsPitching', 'doublesPitching', 'triplesPitching', 'homeRunsPitching', 'strikeOutsPitching',\n                'baseOnBallsPitching', 'intentionalWalksPitching', 'hitsPitching', 'hitByPitchPitching', 'caughtStealingPitching', 'stolenBasesPitching',\n                'saveOpportunities', 'earnedRuns', 'balls', 'strikes', 'hitBatsmen', 'balks', 'wildPitches', 'pickoffsPitching', 'rbiPitching',\n                'gamesFinishedPitching', 'inheritedRunners', 'inheritedRunnersScored', 'catchersInterferencePitching', 'sacBuntsPitching', 'sacFliesPitching',\n                'saves', 'holds', 'blownSaves']\n\npitchPCA = playerBoxScores[pitchingFeat].fillna(0)\n\nn_comp, var_ratio = [], []\n\nfor i in range(1, 21):\n    pca = PCA(n_components = i, random_state = 1)\n    pca.fit(pitchPCA)\n    \n    n_comp.append(i)\n    var_ratio.append(pca.explained_variance_ratio_.sum())\n    \nplt.plot(n_comp, var_ratio)\n#plt.show()\n\n# n = 4","metadata":{"execution":{"iopub.status.busy":"2021-07-31T14:56:53.509485Z","iopub.execute_input":"2021-07-31T14:56:53.51251Z","iopub.status.idle":"2021-07-31T14:57:15.98836Z","shell.execute_reply.started":"2021-07-31T14:56:53.512445Z","shell.execute_reply":"2021-07-31T14:57:15.987295Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pitch_pca = PCA(n_components = 4, random_state = 1)\npitch_pca.fit(playerBoxScores[pitchingFeat].fillna(0))","metadata":{"execution":{"iopub.status.busy":"2021-07-31T14:57:15.990006Z","iopub.execute_input":"2021-07-31T14:57:15.990446Z","iopub.status.idle":"2021-07-31T14:57:16.756169Z","shell.execute_reply.started":"2021-07-31T14:57:15.990402Z","shell.execute_reply":"2021-07-31T14:57:16.754905Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pitchPCAdf = pd.concat([playerBoxScores[playerBoxScores[\"gamesPlayedPitching\"] == 1][\"date_playerId\"], playerBoxScores[playerBoxScores[\"gamesPlayedPitching\"] == 1][pitchingFeat]], axis = 1)\npitchPCAdf = pitchPCAdf.fillna(0).reset_index(drop = True)","metadata":{"execution":{"iopub.status.busy":"2021-07-31T14:57:16.7581Z","iopub.execute_input":"2021-07-31T14:57:16.758805Z","iopub.status.idle":"2021-07-31T14:57:16.832407Z","shell.execute_reply.started":"2021-07-31T14:57:16.758759Z","shell.execute_reply":"2021-07-31T14:57:16.831313Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pitch_df = pd.concat([pitchPCAdf.iloc[:, 0], pd.DataFrame(pitch_pca.fit_transform(pitchPCAdf.iloc[:, 1:]))], axis = 1)\npitch_df = pitch_df.rename(columns = {0: \"pitch0\", 1: \"pitch1\", 2: \"pitch2\", 3: \"pitch4\"})\npitch_df","metadata":{"execution":{"iopub.status.busy":"2021-07-31T14:57:16.833996Z","iopub.execute_input":"2021-07-31T14:57:16.834414Z","iopub.status.idle":"2021-07-31T14:57:17.051776Z","shell.execute_reply.started":"2021-07-31T14:57:16.834369Z","shell.execute_reply":"2021-07-31T14:57:17.050614Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Games\n\n- For games, we dropped highly correlated columns and leave those necessary to include game results into the model.","metadata":{}},{"cell_type":"code","source":"fileDir = \"../input/mlbcsv\"\ngames = pd.read_csv(fileDir + \"/games.csv\").iloc[:, 1:]\n\ngames.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-31T14:57:17.053612Z","iopub.execute_input":"2021-07-31T14:57:17.059843Z","iopub.status.idle":"2021-07-31T14:57:17.169012Z","shell.execute_reply.started":"2021-07-31T14:57:17.05977Z","shell.execute_reply":"2021-07-31T14:57:17.167907Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"games.columns","metadata":{"execution":{"iopub.status.busy":"2021-07-31T14:57:17.170476Z","iopub.execute_input":"2021-07-31T14:57:17.170927Z","iopub.status.idle":"2021-07-31T14:57:17.178849Z","shell.execute_reply.started":"2021-07-31T14:57:17.170884Z","shell.execute_reply":"2021-07-31T14:57:17.17755Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"games_corr = games[[col for col in games.columns if \"home\" in col or \"away\" in col]].corr().abs()\nhigh_corr_cols(games_corr)\n\n# drop all wins/losses and use pct instead","metadata":{"execution":{"iopub.status.busy":"2021-07-31T14:57:17.18045Z","iopub.execute_input":"2021-07-31T14:57:17.181281Z","iopub.status.idle":"2021-07-31T14:57:17.207301Z","shell.execute_reply.started":"2021-07-31T14:57:17.18121Z","shell.execute_reply":"2021-07-31T14:57:17.205277Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"games_df = games[[\"gameDate\", \"homeId\", \"homeWinPct\", \"homeWinner\", \"homeScore\", \"awayId\", \"awayWinPct\", \"awayWinner\", \"awayScore\"]]\n\ndel games\n\ngames_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-31T14:57:17.208817Z","iopub.execute_input":"2021-07-31T14:57:17.209452Z","iopub.status.idle":"2021-07-31T14:57:17.231343Z","shell.execute_reply.started":"2021-07-31T14:57:17.209409Z","shell.execute_reply":"2021-07-31T14:57:17.230084Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Standings","metadata":{}},{"cell_type":"code","source":"fileDir = \"../input/mlbcsv\"\nstandings = pd.read_csv(fileDir + \"/standings.csv\").iloc[:, 1:]\n\nstandings = standings[[\"season\",\"gameDate\", \"teamId\", \"streakCode\", \"divisionRank\", \"leagueRank\", \"wins\", \"losses\", \n                       \"pct\", \"divisionChamp\", \"divisionLeader\"]]\nstandings.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-31T14:57:17.233091Z","iopub.execute_input":"2021-07-31T14:57:17.233758Z","iopub.status.idle":"2021-07-31T14:57:17.33483Z","shell.execute_reply.started":"2021-07-31T14:57:17.233702Z","shell.execute_reply":"2021-07-31T14:57:17.33365Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"divisionRank_byDates = {}\n\nfor index in standings.index:\n    gamedate = standings.at[index,\"gameDate\"][:4] + standings.at[index,\"gameDate\"][5:7] + standings.at[index,\"gameDate\"][8:10]\n    teamid = standings.at[index,\"teamId\"]\n    \n    streakCode = standings.at[index,\"streakCode\"]\n    divisionRank = standings.at[index,\"divisionRank\"]\n    leagueRank = standings.at[index,\"leagueRank\"]\n    pct = standings.at[index,\"pct\"]\n    divisionChamp = standings.at[index,\"divisionChamp\"]\n    divisionLeader = standings.at[index,\"divisionLeader\"]\n    \n    \n    if teamid not in divisionRank_byDates.keys():\n        divisionRank_byDates[teamid] = {gamedate:[streakCode,divisionRank,leagueRank,pct,divisionChamp,divisionLeader]}\n\n    else:\n        divisionRank_byDates[teamid][gamedate] = [streakCode,divisionRank,leagueRank,pct,divisionChamp,divisionLeader]","metadata":{"execution":{"iopub.status.busy":"2021-07-31T14:57:17.336266Z","iopub.execute_input":"2021-07-31T14:57:17.336731Z","iopub.status.idle":"2021-07-31T14:57:18.468789Z","shell.execute_reply.started":"2021-07-31T14:57:17.336637Z","shell.execute_reply":"2021-07-31T14:57:18.467721Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Target","metadata":{}},{"cell_type":"code","source":"fileDir = \"../input/mlbcsv\"\ntarget = pd.read_csv(fileDir + \"/target.csv\").iloc[:, 1:]\n\ntarget.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-31T14:57:18.4704Z","iopub.execute_input":"2021-07-31T14:57:18.470873Z","iopub.status.idle":"2021-07-31T14:57:23.13494Z","shell.execute_reply.started":"2021-07-31T14:57:18.470828Z","shell.execute_reply":"2021-07-31T14:57:23.13377Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"target = date_playerId(target, \"engagementMetricsDate\")\ntarget.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-31T14:57:23.136322Z","iopub.execute_input":"2021-07-31T14:57:23.136772Z","iopub.status.idle":"2021-07-31T14:57:30.830409Z","shell.execute_reply.started":"2021-07-31T14:57:23.136727Z","shell.execute_reply":"2021-07-31T14:57:30.82932Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"target_df = target[target[\"playerId\"].isin(playersForTest[\"playerId\"])]\n\ndel target\n\ntarget_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-31T14:57:30.832213Z","iopub.execute_input":"2021-07-31T14:57:30.832718Z","iopub.status.idle":"2021-07-31T14:57:31.297243Z","shell.execute_reply.started":"2021-07-31T14:57:30.832675Z","shell.execute_reply":"2021-07-31T14:57:31.295821Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"targetShift = pd.DataFrame()\n\nfor playerId in target_df[\"playerId\"].unique():\n    \n    df = target_df[target_df[\"playerId\"] == playerId]\n    df = df.set_index(\"engagementMetricsDate\").shift(1).dropna()\n    \n    targetShift = targetShift.append(df)\n\ntargetShift = targetShift.reset_index()\ntargetShift.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-31T14:57:31.299051Z","iopub.execute_input":"2021-07-31T14:57:31.299479Z","iopub.status.idle":"2021-07-31T14:59:15.996545Z","shell.execute_reply.started":"2021-07-31T14:57:31.299437Z","shell.execute_reply":"2021-07-31T14:59:15.995455Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Delete train and target_df","metadata":{}},{"cell_type":"code","source":"del target_df","metadata":{"execution":{"iopub.status.busy":"2021-07-31T14:59:34.775253Z","iopub.execute_input":"2021-07-31T14:59:34.775633Z","iopub.status.idle":"2021-07-31T14:59:34.783493Z","shell.execute_reply.started":"2021-07-31T14:59:34.775569Z","shell.execute_reply":"2021-07-31T14:59:34.782Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Merge All","metadata":{}},{"cell_type":"code","source":"# Merge Season\nseasons_df = seasons_df.drop_duplicates()\ntargetShift = targetShift.merge(seasons_df.drop_duplicates(), how = \"left\", left_on = \"engagementMetricsDate\", right_on = \"date\").drop(columns = \"date\")\n\n# Merge Awards\ntargetShift = targetShift.merge(awards, how = \"left\", on = \"date_playerId\").fillna(0).drop(columns = [\"playerId_y\"])\n\n# Merge playerTwitter\nplayerTwitter_df = date_playerId(playerTwitter.reset_index(), \"index\")\nplayerTwitter_df = playerTwitter_df.drop(columns = [\"index\", \"playerId\"])\n\ntargetShift = targetShift.merge(playerTwitter_df, how = \"left\", on=\"date_playerId\")\n\nfor index in targetShift.index:\n    if pd.isna(targetShift.at[index,\"numberOfFollowers\"]):\n        targetShift.at[index,\"numberOfFollowers\"] = 0\n\n# Merge boxScore\ntargetShift = targetShift.merge(pitch_df, how = \"left\", on = \"date_playerId\").fillna(0)\ntargetShift = targetShift.merge(bat_df, how = \"left\", on = \"date_playerId\").fillna(0)\ntargetShift = targetShift.merge(playerBoxScores[[\"date_playerId\", \"home\", \"positionCode\"]], how = \"left\", on = \"date_playerId\")\n\n# Merge rosters\ntargetShift = targetShift.merge(rosters, how = \"left\", on = \"date_playerId\") #.drop(columns = [\"playerId_y\"])\n\n# standings\ntargetShift = targetShift.merge(standings, how = \"left\", left_on = [\"engagementMetricsDate\", \"teamId\"], right_on = [\"gameDate\", \"teamId\"])\n\n# Merge games\ntargetShift = targetShift.merge(games_df, how = \"left\", left_on = [\"gameDate\", \"teamId\"], right_on = [\"gameDate\", \"homeId\"])\ntargetShift.head()\n\n# Merge teamTwitterFollowers\ntargetShift = targetShift.merge(teamTwitter, how = \"left\", left_on = [\"gameDate\", \"teamId\"], right_on = [\"date\", \"teamId\"])\n\nfor index in targetShift.index:\n    if pd.isna(targetShift.at[index,\"teamFollowers\"]):\n        targetShift.at[index,\"teamFollowers\"] = 0.\n\ntargetShift.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-31T14:59:40.304047Z","iopub.execute_input":"2021-07-31T14:59:40.304466Z","iopub.status.idle":"2021-07-31T15:02:38.957016Z","shell.execute_reply.started":"2021-07-31T14:59:40.304428Z","shell.execute_reply":"2021-07-31T15:02:38.9559Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for index in targetShift.index:\n    if pd.isna(targetShift.at[index,\"teamId\"]):\n        targetShift.at[index,\"teamId\"] = 0\n\nfor index in targetShift.index:\n    teamId        = targetShift.at[index,\"teamId\"]\n    divisionRank  = targetShift.at[index,\"divisionRank\"]\n    leagueRank    = targetShift.at[index,\"leagueRank\"]\n    pct           = targetShift.at[index,\"pct\"]\n\n    date = targetShift.at[index,\"date_playerId\"][:8]\n    \n    if pd.isna(divisionRank):\n        targetShift.at[index,\"divisionRank\"] = 6\n        \n    if pd.isna(leagueRank):\n        targetShift.at[index,\"leagueRank\"] = 16    \n\n    if pd.isna(leagueRank):\n        targetShift.at[index,\"pct\"] = -1  \n        \n    if teamId in divisionRank_byDates.keys():\n        if date in divisionRank_byDates[teamId].keys():\n            targetShift.at[index,\"streakCode\"]     = divisionRank_byDates[teamId][date][0]\n            targetShift.at[index,\"divisionRank\"]   = int(divisionRank_byDates[teamId][date][1])\n            targetShift.at[index,\"leagueRank\"]     = int(divisionRank_byDates[teamId][date][2])\n            targetShift.at[index,\"pct\"]            = float(divisionRank_byDates[teamId][date][3])\n\ntargetShift.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-31T15:02:38.960075Z","iopub.execute_input":"2021-07-31T15:02:38.960504Z","iopub.status.idle":"2021-07-31T15:09:54.952477Z","shell.execute_reply.started":"2021-07-31T15:02:38.960474Z","shell.execute_reply":"2021-07-31T15:09:54.951529Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"currentPlayer = targetShift.at[0,\"playerId\"]\n\ncurrentdivisionRank = 6.\ncurrentleagueRank = 16.\ncurrentpct = -1.\n\nfor index in targetShift.index:\n    playerid = targetShift.at[index,\"playerId\"]\n    #date = targetShift.at[index,\"date_playerId\"][:8]\n    divisionRank = float(targetShift.at[index,\"divisionRank\"])\n    leagueRank = float(targetShift.at[index,\"leagueRank\"])\n    pct = float(targetShift.at[index,\"pct\"])\n    #teamId = targetShift.at[index,\"teamId\"]\n    \n    if currentPlayer == playerid:\n        \n        if currentdivisionRank != divisionRank:\n            \n            # when standing update happens\n            if divisionRank < 6:\n                currentdivisionRank = divisionRank\n                targetShift.at[index,\"divisionRank\"] = currentdivisionRank\n                \n            # updating all 6 after / when divisionRank = 6\n            elif currentdivisionRank <6:\n                targetShift.at[index,\"divisionRank\"] = currentdivisionRank\n                \n        if currentleagueRank != leagueRank:\n            \n            # when standing update happens\n            if leagueRank < 16:\n                currentleagueRank = leagueRank\n                targetShift.at[index,\"leagueRank\"] = currentleagueRank\n                \n            # updating all 16 after / when divisionRank = 16\n            elif currentleagueRank <16:\n                targetShift.at[index,\"leagueRank\"] = currentleagueRank\n                \n        if currentpct != -1:\n            \n            # when standing update happens\n            if pct > -1:\n                currentpct = pct\n                targetShift.at[index,\"pct\"] = currentpct\n                \n            # updating all 16 after / when divisionRank = 16\n            elif currentpct > -1:\n                targetShift.at[index,\"pct\"] = currentpct\n                \n    else:\n        currentPlayer = playerid\n        currentdivisionRank = 6\n        currentleagueRank = 16\n        currentpct = -1","metadata":{"execution":{"iopub.status.busy":"2021-07-31T15:09:54.954985Z","iopub.execute_input":"2021-07-31T15:09:54.955364Z","iopub.status.idle":"2021-07-31T15:13:06.997516Z","shell.execute_reply.started":"2021-07-31T15:09:54.955324Z","shell.execute_reply":"2021-07-31T15:13:06.996412Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for index in targetShift.index:\n    \n    # if record is empty for streakCode, then replace them to 0\n    if pd.isna(targetShift.at[index,\"streakCode\"]):\n        targetShift.at[index,\"streakCode\"] = 0.\n    \n    else:\n        if \"W\" in targetShift.at[index,\"streakCode\"]:\n            targetShift.at[index,\"streakCode\"] = float(targetShift.at[index,\"streakCode\"][1])\n            \n        elif \"L\" in targetShift.at[index,\"streakCode\"]:\n            targetShift.at[index,\"streakCode\"] = -(float(targetShift.at[index,\"streakCode\"][1]))","metadata":{"execution":{"iopub.status.busy":"2021-07-31T15:13:06.999662Z","iopub.execute_input":"2021-07-31T15:13:07.000113Z","iopub.status.idle":"2021-07-31T15:15:02.671784Z","shell.execute_reply.started":"2021-07-31T15:13:07.000056Z","shell.execute_reply":"2021-07-31T15:15:02.670741Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"targetShift = targetShift[['engagementMetricsDate', 'date_playerId','teamId', 'seasonIdx', 'isAwarded',\n                           'cumulativeAwardScore', 'numberOfFollowers', 'teamFollowers',\n                           'pitch0', 'pitch1','pitch2', 'pitch4',\n                           'bat0', 'bat1', 'bat2', 'bat3', 'bat4', 'bat5', 'bat6',\n                            'streakCode', 'divisionRank', 'leagueRank','wins', 'losses', 'pct', 'divisionChamp', 'divisionLeader',\n                           'homeWinPct', 'homeWinner', 'homeScore',\n                           'awayWinPct', 'awayScore',\n                           'target1', 'target2', 'target3','target4' ]]   #'home','awayWinner'\n\ntargetShift.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-31T15:16:57.697067Z","iopub.execute_input":"2021-07-31T15:16:57.697442Z","iopub.status.idle":"2021-07-31T15:16:58.88687Z","shell.execute_reply.started":"2021-07-31T15:16:57.697411Z","shell.execute_reply":"2021-07-31T15:16:58.885754Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for index in targetShift.index:\n    homeWinPct = targetShift.at[index,\"homeWinPct\"]\n    homeWinner = targetShift.at[index,\"homeWinner\"]\n    homeScore = targetShift.at[index,\"homeScore\"]\n    awayWinPct = targetShift.at[index,\"awayWinPct\"]\n    awayScore = targetShift.at[index,\"awayScore\"]\n    seasonIdx = targetShift.at[index,\"seasonIdx\"]\n    \n    streakCode = targetShift.at[index,\"streakCode\"]\n    divisionRank = targetShift.at[index,\"divisionRank\"]\n    leagueRank = targetShift.at[index,\"leagueRank\"]\n    wins = targetShift.at[index,\"wins\"]\n    losses = targetShift.at[index,\"losses\"]\n    pct = targetShift.at[index,\"pct\"] \n    divisionChamp = targetShift.at[index,\"divisionChamp\"]\n    divisionLeader = targetShift.at[index,\"divisionLeader\"]\n    # off-season\n    #if seasonIdx == 0:\n    \n    if pd.isna(homeWinPct):\n        targetShift.at[index,\"homeWinPct\"]= -1.\n\n    if pd.isna(homeWinner):\n        targetShift.at[index,\"homeWinner\"]= -1.   \n    elif homeWinner == False:\n        targetShift.at[index,\"homeWinner\"] = 0. \n    elif homeWinner == True:\n        targetShift.at[index,\"homeWinner\"] = 1.\n\n    if pd.isna(homeScore):\n        targetShift.at[index,\"homeScore\"] = -1.\n    if pd.isna(awayWinPct):\n        targetShift.at[index,\"awayWinPct\"] = -1. \n    if pd.isna(awayScore):\n        targetShift.at[index,\"awayScore\"] = -1.   \n    if pd.isna(streakCode):\n        targetShift.at[index,\"streakCode\"] = 0.\n    if pd.isna(divisionRank):\n        targetShift.at[index,\"divisionRank\"] = 6.    \n    if pd.isna(leagueRank):\n        targetShift.at[index,\"leagueRank\"] = 16. \n    if pd.isna(wins):\n        targetShift.at[index,\"wins\"] = -1.       \n    if pd.isna(losses):\n        targetShift.at[index,\"losses\"] = -1.\n    if pd.isna(pct):\n        targetShift.at[index,\"pct\"] = -1.\n\n    if divisionChamp == True:\n        targetShift.at[index,\"divisionChamp\"] = 1.\n    else:\n        targetShift.at[index,\"divisionChamp\"] = 0.\n        \n    if divisionLeader == True:\n        targetShift.at[index,\"divisionLeader\"] = 1.\n    else:\n        targetShift.at[index,\"divisionLeader\"] = 0.\ntargetShift","metadata":{"execution":{"iopub.status.busy":"2021-07-31T15:16:58.888858Z","iopub.execute_input":"2021-07-31T15:16:58.889257Z","iopub.status.idle":"2021-07-31T15:32:26.111695Z","shell.execute_reply.started":"2021-07-31T15:16:58.889217Z","shell.execute_reply":"2021-07-31T15:32:26.110463Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#targetShift.to_csv(\"ANN_7_July30th.csv\", index=False)","metadata":{"execution":{"iopub.status.busy":"2021-07-31T15:32:26.114408Z","iopub.execute_input":"2021-07-31T15:32:26.114923Z","iopub.status.idle":"2021-07-31T15:32:26.119967Z","shell.execute_reply.started":"2021-07-31T15:32:26.114878Z","shell.execute_reply":"2021-07-31T15:32:26.118424Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Modeling\n\n- We trained simple NN model on the sampled dataset from the train dataset created above.","metadata":{}},{"cell_type":"code","source":"maxplayerFollowers = targetShift[\"numberOfFollowers\"].max()\nmaxteamFollowers = targetShift[\"teamFollowers\"].max()\nprint(maxplayerFollowers,maxteamFollowers)","metadata":{"execution":{"iopub.status.busy":"2021-07-31T15:32:26.12242Z","iopub.execute_input":"2021-07-31T15:32:26.122968Z","iopub.status.idle":"2021-07-31T15:32:26.14242Z","shell.execute_reply.started":"2021-07-31T15:32:26.122916Z","shell.execute_reply":"2021-07-31T15:32:26.14034Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"targetShift[\"numberOfFollowers\"] = ( targetShift[\"numberOfFollowers\"] / maxplayerFollowers ) * 100\ntargetShift[\"teamFollowers\"] = ( targetShift[\"teamFollowers\"] / maxteamFollowers ) * 100\ntargetShift.rename({'numberOfFollowers': 'norm_playerFollowers', 'teamFollowers': 'norm_teamFollowers'}, axis=1, inplace=True)\ntargetShift.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-31T15:32:26.144519Z","iopub.execute_input":"2021-07-31T15:32:26.145078Z","iopub.status.idle":"2021-07-31T15:32:26.205806Z","shell.execute_reply.started":"2021-07-31T15:32:26.145024Z","shell.execute_reply":"2021-07-31T15:32:26.204395Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### ANN approach","metadata":{}},{"cell_type":"code","source":"train = targetShift[targetShift[\"engagementMetricsDate\"] < \"2021-04-25\"]\ntest = targetShift[targetShift[\"engagementMetricsDate\"] >= \"2021-04-25\"]\nprint(len(train), len(test))\n\ndel targetShift","metadata":{"execution":{"iopub.status.busy":"2021-07-31T15:32:26.207791Z","iopub.execute_input":"2021-07-31T15:32:26.208281Z","iopub.status.idle":"2021-07-31T15:32:26.820723Z","shell.execute_reply.started":"2021-07-31T15:32:26.208237Z","shell.execute_reply":"2021-07-31T15:32:26.819517Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"n_sample = 500000\ntrain_sample = train.sample(n = n_sample)\n\nX_train = train_sample[['isAwarded', 'cumulativeAwardScore', 'norm_playerFollowers',\n                        'norm_teamFollowers', 'pitch0', 'pitch1', 'pitch2', 'pitch4', 'bat0',\n                        'bat1', 'bat2', 'bat3', 'bat4', 'bat5', 'bat6', 'streakCode',\n                        'divisionRank', 'leagueRank', 'wins', 'losses', 'pct', 'divisionChamp',\n                        'divisionLeader','homeWinPct', 'homeWinner', 'homeScore', 'awayWinPct', 'awayScore']].fillna(0)\nY_train = train_sample[[\"target1\", \"target2\", \"target3\", \"target4\"]]\n\nprint(\"Train Data Shape : {}\".format(X_train.shape))\nprint(\"Train Label Shape : {}\".format(Y_train.shape))","metadata":{"execution":{"iopub.status.busy":"2021-07-31T15:32:26.822436Z","iopub.execute_input":"2021-07-31T15:32:26.823125Z","iopub.status.idle":"2021-07-31T15:32:27.955741Z","shell.execute_reply.started":"2021-07-31T15:32:26.823079Z","shell.execute_reply":"2021-07-31T15:32:27.954688Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"feature_size = X_train.shape[1]\n\ndef build_network(input_shape=(feature_size,)):\n    model = models.Sequential()\n    model.add(layers.Dense(28, activation='relu', input_shape=input_shape))\n    model.add(layers.Dropout(0.2))\n    model.add(layers.Dense(28, activation='relu'))\n    model.add(layers.Dropout(0.2))\n    model.add(layers.Dense(28, activation='relu'))\n    model.add(layers.Dropout(0.2))\n    model.add(layers.Dense(14, activation='relu'))\n    model.add(layers.Dropout(0.2))\n    model.add(layers.Dense(1))\n    model.compile(optimizer='adam', loss='mse', metrics=['mae'])\n    return model","metadata":{"execution":{"iopub.status.busy":"2021-07-31T15:32:27.961811Z","iopub.execute_input":"2021-07-31T15:32:27.964381Z","iopub.status.idle":"2021-07-31T15:32:27.977602Z","shell.execute_reply.started":"2021-07-31T15:32:27.964337Z","shell.execute_reply":"2021-07-31T15:32:27.976414Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model1 = build_network((feature_size, ))\nmodel2 = build_network((feature_size, ))\nmodel3 = build_network((feature_size, ))\nmodel4 = build_network((feature_size, ))\nmodel1.summary()","metadata":{"execution":{"iopub.status.busy":"2021-07-31T15:32:27.983099Z","iopub.execute_input":"2021-07-31T15:32:27.986159Z","iopub.status.idle":"2021-07-31T15:32:30.652897Z","shell.execute_reply.started":"2021-07-31T15:32:27.986082Z","shell.execute_reply":"2021-07-31T15:32:30.651766Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model1.fit(X_train.values, Y_train.iloc[:, 0].values, epochs=20)","metadata":{"execution":{"iopub.status.busy":"2021-07-31T15:39:40.503441Z","iopub.execute_input":"2021-07-31T15:39:40.503854Z","iopub.status.idle":"2021-07-31T15:52:16.010328Z","shell.execute_reply.started":"2021-07-31T15:39:40.503823Z","shell.execute_reply":"2021-07-31T15:52:16.009036Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model2.fit(X_train.values, Y_train.iloc[:, 1].values, epochs=20)","metadata":{"execution":{"iopub.status.busy":"2021-07-31T15:52:39.68186Z","iopub.execute_input":"2021-07-31T15:52:39.682248Z","iopub.status.idle":"2021-07-31T16:05:18.015601Z","shell.execute_reply.started":"2021-07-31T15:52:39.682217Z","shell.execute_reply":"2021-07-31T16:05:18.014561Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model3.fit(X_train.values, Y_train.iloc[:, 2].values, epochs=20)","metadata":{"execution":{"iopub.status.busy":"2021-07-31T16:05:36.580624Z","iopub.execute_input":"2021-07-31T16:05:36.581062Z","iopub.status.idle":"2021-07-31T16:18:08.043136Z","shell.execute_reply.started":"2021-07-31T16:05:36.58103Z","shell.execute_reply":"2021-07-31T16:18:08.04205Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model4.fit(X_train.values, Y_train.iloc[:, 3].values, epochs=20)","metadata":{"execution":{"iopub.status.busy":"2021-07-31T16:18:08.045113Z","iopub.execute_input":"2021-07-31T16:18:08.045534Z","iopub.status.idle":"2021-07-31T16:30:50.464568Z","shell.execute_reply.started":"2021-07-31T16:18:08.045492Z","shell.execute_reply":"2021-07-31T16:30:50.463495Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Predictions\n\n- From the models and test set created above, we will make a prediction and save the result into dataframe.","metadata":{}},{"cell_type":"code","source":"test_id = test[\"date_playerId\"]\nX_test = test[['isAwarded', 'cumulativeAwardScore', 'norm_playerFollowers',\n               'norm_teamFollowers', 'pitch0', 'pitch1', 'pitch2', 'pitch4', 'bat0',\n               'bat1', 'bat2', 'bat3', 'bat4', 'bat5', 'bat6', 'streakCode',\n               'divisionRank', 'leagueRank', 'wins', 'losses', 'pct', 'divisionChamp',\n               'divisionLeader','homeWinPct', 'homeWinner', 'homeScore', 'awayWinPct', 'awayScore']].fillna(0)","metadata":{"execution":{"iopub.status.busy":"2021-07-31T16:37:43.992326Z","iopub.execute_input":"2021-07-31T16:37:43.992868Z","iopub.status.idle":"2021-07-31T16:37:44.01035Z","shell.execute_reply.started":"2021-07-31T16:37:43.992837Z","shell.execute_reply":"2021-07-31T16:37:44.009122Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Y_pred1 = model1.predict(X_test.values)\nY_pred2 = model2.predict(X_test.values)\nY_pred3 = model3.predict(X_test.values)\nY_pred4 = model4.predict(X_test.values)","metadata":{"execution":{"iopub.status.busy":"2021-07-31T16:37:44.477562Z","iopub.execute_input":"2021-07-31T16:37:44.477999Z","iopub.status.idle":"2021-07-31T16:37:45.786827Z","shell.execute_reply.started":"2021-07-31T16:37:44.477968Z","shell.execute_reply":"2021-07-31T16:37:45.785694Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"prediction_df = pd.DataFrame()\nprediction_df[\"date_playerId\"] = test_id\nprediction_df[\"target1\"] = Y_pred1\nprediction_df[\"target2\"] = Y_pred2\nprediction_df[\"target3\"] = Y_pred3\nprediction_df[\"target4\"] = Y_pred4\nprediction_df = prediction_df.reset_index(drop = True)\nprediction_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-31T16:41:43.482956Z","iopub.execute_input":"2021-07-31T16:41:43.483317Z","iopub.status.idle":"2021-07-31T16:41:43.509993Z","shell.execute_reply.started":"2021-07-31T16:41:43.483286Z","shell.execute_reply":"2021-07-31T16:41:43.508742Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import mlb\nenv = mlb.make_env() # initialize the environment\niter_test = env.iter_test() # iterator which loops over each date in test set\n\nfor (test_df, prediction_df) in iter_test:\n    env.predict(prediction_df)","metadata":{"execution":{"iopub.status.busy":"2021-07-31T16:42:42.6512Z","iopub.execute_input":"2021-07-31T16:42:42.651685Z","iopub.status.idle":"2021-07-31T16:42:44.155596Z","shell.execute_reply.started":"2021-07-31T16:42:42.651653Z","shell.execute_reply":"2021-07-31T16:42:44.154372Z"},"trusted":true},"execution_count":null,"outputs":[]}]}