{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"Please open this kernel with maximum viewing area. You can either open it on a wide monitor or simply click on the burger icon to remove the left hand side panel where you have competition, code, dataset, etc..\n\n\n\n# ⛄ Introduction\n\n> In this competition, you’ll predict how fans engage with MLB players’ digital content on a daily basis for a future date range. You’ll have access to player performance data, social media data, and team factors like market size. Successful models will provide new insights into what signals most strongly correlate with and influence engagement.\n\n## ❄️ Problem Statement\n\nIn this competition, the task is to \"forecast\" four different measures of engagement (`target1` - `target4`) for a subset of MLB players who are active in the 2021 season. \n\nSince we have to forecast, given an input date `d` we need to predict `target1` - `target4` for the date `d+1`.\n\n## 📜 Training Data\n\nWe have both \"static\" files and \"daily\" data: \n\n* Static files - The files that are static are `players.csv`, `teams.csv`, `seasons.csv`, and `awards.csv`. These files don't change with time or to put it simply there is no continuous date column in these `csv` files. \n\n* Daily file - We have a `train.csv` file which is grouped by day or simply put there is a continuous date column. The first row is associated with the date `01-01-2018` and last row is associated with `some-date`.
\n\n![\"Weights](\"https://i.imgur.com/gb6B4ig.png\")
\n\n# 💎 W&B Tables\n\nWB Tables accelerate the ML development lifecycle by giving users the ability to rapidly extract meaningful insights from tabular data. The WB Table Visualizer provides an interactive interface to perform powerful analytics functions like grouping, joining, and creating custom fields while simultaneously supporting rich media annotations such as bounding boxes and segmentation masks. \n\nWB Tables is designed \"generically\" to work well for a wide range of use cases - from analyzing intermediate data transformations to reviewing model predictions - while being directly integrated directly into the WB UI dashboard, allowing users to learn, adapt, and improve their models effectively and efficiently.\n\nLearn more about W&B Tables [here](https://docs.wandb.ai/guides/data-vis). Note that this feature is still work in progress and would love if you use and and send over feedbacks. \n\n## ♣️ About this kernel\n\nIn this kernel, we will go through each csv files and perform meaningful EDA. We will use W&B Tables for quick EDA and W&B Custom Charts for more complex visualizations. We will also use conventional matplotlib to get some insights. \n\nThis is a work in progress. :D","metadata":{}},{"cell_type":"markdown","source":"# ⚙️ Imports and Setups","metadata":{}},{"cell_type":"code","source":"!pip install --upgrade -q wandb","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2021-06-18T01:27:40.438912Z","iopub.execute_input":"2021-06-18T01:27:40.439532Z","iopub.status.idle":"2021-06-18T01:27:53.363596Z","shell.execute_reply.started":"2021-06-18T01:27:40.439429Z","shell.execute_reply":"2021-06-18T01:27:53.362469Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\nos.environ[\"WANDB_SILENT\"] = \"true\"\nimport gc\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\nimport wandb\nwandb.login()","metadata":{"execution":{"iopub.status.busy":"2021-06-18T01:28:06.606883Z","iopub.execute_input":"2021-06-18T01:28:06.607402Z","iopub.status.idle":"2021-06-18T01:28:09.667891Z","shell.execute_reply.started":"2021-06-18T01:28:06.607367Z","shell.execute_reply":"2021-06-18T01:28:09.666737Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> Find you API key at: https://wandb.ai/authorize","metadata":{}},{"cell_type":"code","source":"CONFIG = dict(\n competition = 'mlb',\n _wandb_kernel = 'ayut',\n infra = \"Kaggle\",\n)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-06-18T01:28:37.182618Z","iopub.execute_input":"2021-06-18T01:28:37.183362Z","iopub.status.idle":"2021-06-18T01:28:37.188021Z","shell.execute_reply.started":"2021-06-18T01:28:37.183322Z","shell.execute_reply":"2021-06-18T01:28:37.187147Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load CSV Files","metadata":{}},{"cell_type":"code","source":"ROOT_PATH = \"/kaggle/input/mlb-player-digital-engagement-forecasting/\"\n\n# training file\ntrain_df = pd.read_csv(f\"{ROOT_PATH}/train.csv\")\n\n# meta files\nplayers_df = pd.read_csv(f\"{ROOT_PATH}/players.csv\")\nteams_df = pd.read_csv(f\"{ROOT_PATH}/teams.csv\")\nseasons_df = pd.read_csv(f\"{ROOT_PATH}/seasons.csv\")\nawards_df = pd.read_csv(f\"{ROOT_PATH}/awards.csv\")","metadata":{"execution":{"iopub.status.busy":"2021-06-18T01:28:41.238529Z","iopub.execute_input":"2021-06-18T01:28:41.238900Z","iopub.status.idle":"2021-06-18T01:30:00.635325Z","shell.execute_reply.started":"2021-06-18T01:28:41.238868Z","shell.execute_reply":"2021-06-18T01:30:00.633676Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# EDA of Static CSV files","metadata":{}},{"cell_type":"markdown","source":"## 1️⃣ players.csv\n\n> #### 📌 Column Names and Descriptions
\n`playerId` - Unique identifier for a player.
\n`playerName` - Name of players. There are 2055 unique player names. \n`DOB` - Player’s date of birth.
\n`mlbDebutDate`
\n`birthCity`
\n`birthStateProvince`
\n`birthCountry`
\n`heightInches`
\n`weight`
\n`primaryPositionCode` - Player’s primary position code, details are here.
\n`primaryPositionName` - player’s primary position, details are here.
\n`playerForTestSetAndFuturePreds` - Boolean, true if player is among those for whom predictions are to be made in test data
","metadata":{}},{"cell_type":"code","source":"player_table = wandb.Table(dataframe=players_df)\n\nrun = wandb.init(project='kaggle-mlb', config=CONFIG)\nwandb.log({'raw_players_table': player_table})\nrun.finish() \nrun","metadata":{"execution":{"iopub.status.busy":"2021-06-17T09:08:07.786597Z","iopub.execute_input":"2021-06-17T09:08:07.787925Z","iopub.status.idle":"2021-06-17T09:08:30.156285Z","shell.execute_reply.started":"2021-06-17T09:08:07.787798Z","shell.execute_reply":"2021-06-17T09:08:30.154620Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### You can also check out the created W&B tables page [here](https://wandb.ai/ayush-thakur/kaggle-mlb/runs/944575zt). \n\n#### 📌 1. How many rows?\n\nThere are 2061 rows in the `players.csv` file.\n\n\n#### 📌 2. Do we have unique player names?\n\nThere are 2055 unique player names. There are 5 player names that occur multiple times. These are Luis Garcia (3), Will Smith (2), Javy Guerra (2), Austin Adams (2), Jose Ramirez (2). \n\n🎳 **Do It Yourself**: Group the `playerName` column and insert new column to the right of the `playerName` column. Go to the Column Setting of this new column and in the cell expression select use `row.count()`, this will give the `count` column. Sort this column in descending order. Scoll though the table and get more useful insights. ","metadata":{}},{"cell_type":"code","source":"run","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-06-17T09:08:43.156608Z","iopub.execute_input":"2021-06-17T09:08:43.157055Z","iopub.status.idle":"2021-06-17T09:08:43.166794Z","shell.execute_reply.started":"2021-06-17T09:08:43.157016Z","shell.execute_reply":"2021-06-17T09:08:43.165488Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> 📎 Let's dig into Luis Garcia more: \nThere's three DOB associated with Luis Garcia, that means they are different individuals. They even have different birth city. One of them comes from New York city. Two of them played at the position with `primaryPositionCode` of 1 and played as pitcher.\n","metadata":{}},{"cell_type":"markdown","source":"#### 📌 4. When were the players born Date of Births\n\nUsing the technique as above we can find that there are only 1670 unique date of births. That means multiple players were born on the same day. Interesting! \n\n5 MLB players are born on `1995-01-17` in particular of which 3 plays as pitcher. ","metadata":{}},{"cell_type":"markdown","source":"#### 📌 5. What are the positions these players play?\n\nThere are 10 unique playing positions in the `players.csv` file. These are - Pitcher, Catcher, Outfilder, First base, Second Base, Third base, Shortstop, outfield, Designated Hitter, and Infield. \n\nFrom the Baseball Positions [wikipedia page](https://en.wikipedia.org/wiki/Baseball_positions) there are total of 9 fielding positions that can be grouped into three groups:\n\n* Outfield (left field, center field, and right field)\n\n* Infield (first base, second base, third base, and shortstop)\n\n* Battery (pitcher and catcher)\n\n🎳 **Do It Yourself**: Head over to the table and group by `primaryPositionName` column. Insert a new column to the left handside of the `playerName` column by clickin on the `Insert 1 left <` from the three dot icon. You will get this three dot by hovering over the column name. Click on the `Column settings` from the three dot icon of the newly created column. In the `cell expression` box input `row.count()`. Sort the hence created `count` column in descending order.\n\n","metadata":{}},{"cell_type":"markdown","source":"\n([Source](https://en.wikipedia.org/wiki/Baseball_positions))\n\nLet's map the `primaryPositionName` to `primaryPositionCode`.\n\n* Pitcher -> 1
\n* Outfielder -> 7 ((left fielder), 8 (center fielder), 9(right fielder)
\n* Catcher -> 2
\n* Second Base -> 4
\n* First Base -> 3
\n* Shortstop -> 6
\n* Third Base -> 5
\n* Outfield -> \"O\"
\n* Designated Hitter -> 10
\n* Infield -> \"I\"
\n\nNote that Designated Hitter is a special role. More on it [here](https://en.wikipedia.org/wiki/Designated_hitter).\n\nWe might want to the `primaryPositionCode` as a feature. ","metadata":{}},{"cell_type":"markdown","source":"#### 📌 6. What's `playerForTestSetAndFuturePreds`?\n\nThese players will be present in the test set and we need to forecast the measure of engagement for them. So how many players are present in the test set?\n\nLet's quickly use the filter feature of the Tables to find it. \n\nDIY: Click on Filter and in the filter expression use the expression `run[\"playerForTestSetAndFuturePreds\"]` and click on apply. \n\n**There are only 1187 players in the test set.**\n\n","metadata":{}},{"cell_type":"markdown","source":"## 2️⃣ seasons.csv\n\n> #### 📌 Column Names and Description
\n`seasonId`: Each year starting from 2017 is a season and includes 2021.
\n`seasonStartDate`
\n`seasonEndDate`
\n`preSeasonStartDate`
\n`preSeasonEndDate`
\n`regularSeasonStartDate`
\n`regularSeasonEndDate`
\n`lastDate1stHalf`
\n`allStarDate`
\n`firstDate2ndHalf`
\n`postSeasonStartDate`
\n`postSeasonEndDate`","metadata":{}},{"cell_type":"code","source":"seasons_table = wandb.Table(dataframe=seasons_df)\n\nrun = wandb.init(project='kaggle-mlb', config=CONFIG)\nwandb.log({'raw_seasons_table': seasons_table})\nrun.finish() \nrun","metadata":{"execution":{"iopub.status.busy":"2021-06-17T09:08:53.007219Z","iopub.execute_input":"2021-06-17T09:08:53.007609Z","iopub.status.idle":"2021-06-17T09:09:08.313672Z","shell.execute_reply.started":"2021-06-17T09:08:53.007576Z","shell.execute_reply":"2021-06-17T09:09:08.312241Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### You can also check out the W&B Tables page [here](https://wandb.ai/ayush-thakur/kaggle-mlb/runs/12fs9zho).\n\n* We know the import start and end dates for 5 seasons - 2017 - 2021
\n* We will forecast the for the season of 2021. We can use this season's data as validation set.
\n* A typical MLB season lasts for approximately 7 months. In the year 2017-2019, the seasons were 7 months long.
\n* The 2020 season was only for 3 months because of Covid-19 pandemic. From the [2020 MLB wikipedia page](https://en.wikipedia.org/wiki/2020_Major_League_Baseball_season), \"The 2020 Major League Baseball season began on July 23 and ended on September 27 with 60 games amidst the ongoing COVID-19 pandemic.\"
\n* The 2021 season is scheduled for 8 months.
\n* In Major League Baseball (MLB), spring training is a series of practices and exhibition games preceding the start of the regular season. This is called pre-season. I would assume the engagement to be lower in pre-season games.
\n* The Major League Baseball \"postseason\" is an elimination tournament held after the conclusion of the Major League Baseball (MLB) regular season.
\n* The Major League Baseball All-Star Game, also known as the \"Midsummer Classic\", is an annual professional baseball game sanctioned by Major League Baseball (MLB) and contested between the all-stars from the American League (AL) and National League (NL). There was no all-star game in 2020.
\n* In 2021, preseason start date and season start date are the same. Wondering why?","metadata":{}},{"cell_type":"markdown","source":"## 3️⃣ teams.csv\n\n> #### 📌 Column Names and Description
\n`id` - teamId
\n`name`
\n`teamName`
\n`teamCode`
\n`shortName`
\n`abbreviation`
\n`locationName`
\n`leagueId`
\n`leagueName`
\n`divisionId`
\n`divisionName`
\n`venueId`
\n`venueName`
","metadata":{}},{"cell_type":"code","source":"teams_table = wandb.Table(dataframe=teams_df)\n\nrun = wandb.init(project='kaggle-mlb')\nwandb.log({'raw_teams_table': teams_table})\nrun.finish() \nrun","metadata":{"execution":{"iopub.status.busy":"2021-06-17T09:09:18.737185Z","iopub.execute_input":"2021-06-17T09:09:18.737733Z","iopub.status.idle":"2021-06-17T09:09:34.022474Z","shell.execute_reply.started":"2021-06-17T09:09:18.737693Z","shell.execute_reply":"2021-06-17T09:09:34.021327Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### You can also check out the W&B Tables page [here](https://wandb.ai/ayush-thakur/kaggle-mlb/runs/1xpejcwd)\n\n#### 📌 1. Number of teams - 30\n#### 📌 2. How many divisons are there? \n\nThere are 6 divisions with 5 teams per division. \n\n🎳 **Do It Yourself**: Head over to the table and group by `divisionId`. \n\n","metadata":{}},{"cell_type":"markdown","source":"#### 📌 3. Number of leagues?\n\nThere are two unique leagueIds - 103 and 104 with 15 teams per id. 103 is associated with American League while 104 is associated with National League. \n\n#### 📌 4. Number of unique playing locations?\n\nThere are 29 unique playing locations. There are two teams from Chicago - Chicago Cubs and Chicago White Sox. \n\n🎳 **Do It Yourself**: Head over to the table and group by `locationName`. \n\n","metadata":{}},{"cell_type":"markdown","source":"## 4️⃣ awards.csv\n\n> #### 📌 Column Names and Description
\n`awardDate` - Date award was given.
\n`awardSeason` - Season award was from.
\n`awardId`
\n`awardName`
\n`playerId` - Unique identifier for a player.
\n`playerName`
\n`awardPlayerTeamId`
","metadata":{}},{"cell_type":"code","source":"awards_table = wandb.Table(dataframe=awards_df)\n\nrun = wandb.init(project='kaggle-mlb', config=CONFIG)\nwandb.log({'raw_awards_table': awards_table})\nrun.finish()\nrun","metadata":{"execution":{"iopub.status.busy":"2021-06-17T09:15:40.422178Z","iopub.execute_input":"2021-06-17T09:15:40.422702Z","iopub.status.idle":"2021-06-17T09:15:57.567031Z","shell.execute_reply.started":"2021-06-17T09:15:40.422654Z","shell.execute_reply":"2021-06-17T09:15:57.565809Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### You can also check out the W&B Tables page [here](https://wandb.ai/ayush-thakur/kaggle-mlb/runs/1wu3uepk)\n\n#### 📌 1. Number of rows - 11256\n#### 📌 2. How many seasons are covered by this csv file?\n\nThere are total of 19 season starting from 1998 - 2017. If you look at the counts column in the Table below, you will find that the number of awards given out every season increased from the past season (2017 is an exception with fewer awards given out compared to 2016). Also it won't be too wrong to assume that the award ceremonies in seasons close to 2017 will have far more impact on the engagement as compared to seasons closer to 1998. ","metadata":{}},{"cell_type":"code","source":"run","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-06-17T09:15:57.569109Z","iopub.execute_input":"2021-06-17T09:15:57.569481Z","iopub.status.idle":"2021-06-17T09:15:57.578834Z","shell.execute_reply.started":"2021-06-17T09:15:57.569446Z","shell.execute_reply":"2021-06-17T09:15:57.577435Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> Scroll through the table above to get more insights like, in the season of 2017, Aaron Judge got the most number of awards (17) followed by Ronald Acuna Jr (16) and Jose Altuve (15). Most number of award were MiLB.com Organization All-Star (216) with `awardId` `MILBORGAS`. Most awards were given out on 20th June (123). \n\nIn the `train.csv` file the start date is `2017-01-01` thus in my opinion the `awardDate` in `awards.csv` file is not that useful. However the number of awards that a player has can be useful. A player with more awards should have more fan base thus driving more digital engagements. \n\n#### 📌 3. Numbers of players in this csv file?\n\nThere are 1692 players in the file. Albert Pujols got 70 awards. \n\n**Do It Yourself**: Head over to the table and group by `playerName` and add a count column. Sort the count column in descending order. \n\n","metadata":{}},{"cell_type":"markdown","source":"# Join all the Static Files\n\nIn order to join the `players.csv`, `team.csv`, `seasons.csv`, and `awards.csv` file together we need to find the common columns. \n\nIn the `awards.csv` file, `playerId` and `awardPlayerTeamId` are common with `playerId` in the `players.csv` file and `id` in the `teams.csv` file respectively. The `awardSeason` in the `awards.csv` file is common with `seasonId` in the `seasons.csv` file. \n\nWith W&B Tables we can easily join the tables by using the concept of \"foreign keys\". Check out the table below. ","metadata":{}},{"cell_type":"code","source":"# Create Tables for each df\nawards_table = wandb.Table(dataframe=awards_df)\nplayers_table = wandb.Table(dataframe=players_df)\nteams_table = wandb.Table(dataframe=teams_df)\nseasons_table = wandb.Table(dataframe=seasons_df)\n\n# Clean up the IDs for easy mapping\ndef cleanIds(mapping):\n def cleanIdsFn(ndx, row):\n res = {}\n for oldKey, newKey in mapping:\n if type(row[oldKey]) in [np.float64]:\n item = row[oldKey].item()\n if not np.isnan(item):\n res[newKey] = str(int(item))\n else:\n res[newKey] = \"\"\n else:\n res[newKey] = str(row[oldKey])\n return res\n return cleanIdsFn\n\nawards_table.add_computed_columns(cleanIds([\n (\"awardId\", \"aId\"),\n (\"awardSeason\", \"season\"),\n (\"playerId\", \"player\"),\n (\"awardPlayerTeamId\", \"team\"),\n]))\nseasons_table.add_computed_columns(cleanIds([(\"seasonId\", \"sId\")]))\nteams_table.add_computed_columns(cleanIds([(\"id\", \"tId\")]))\nplayers_table.add_computed_columns(cleanIds([(\"playerId\", \"pId\")]))\n\n# Declare the relationship between \"awards\" and the other tables\nawards_table.set_fk(\"season\", seasons_table, \"sId\")\nawards_table.set_fk(\"player\", players_table, \"pId\")\nawards_table.set_fk(\"team\", teams_table, \"tId\")","metadata":{"execution":{"iopub.status.busy":"2021-06-18T01:40:23.132905Z","iopub.execute_input":"2021-06-18T01:40:23.133334Z","iopub.status.idle":"2021-06-18T01:40:26.527891Z","shell.execute_reply.started":"2021-06-18T01:40:23.133301Z","shell.execute_reply":"2021-06-18T01:40:26.526819Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"run = wandb.init(project='kaggle-mlb', config=CONFIG)\nwandb.log({'joined_static_table': awards_table})\nrun.finish()\nrun","metadata":{"execution":{"iopub.status.busy":"2021-06-18T01:41:05.319798Z","iopub.execute_input":"2021-06-18T01:41:05.320159Z","iopub.status.idle":"2021-06-18T01:41:24.074609Z","shell.execute_reply.started":"2021-06-18T01:41:05.320128Z","shell.execute_reply":"2021-06-18T01:41:24.073610Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### You can also check out the W&B Tables page [here](https://wandb.ai/ayush-thakur/kaggle-mlb/runs/3qw2z9p3)","metadata":{}},{"cell_type":"markdown","source":"# WORK IN PROGRESS. :).","metadata":{}}]}