{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"\n
Major League Baseball
\n\n\nA player hits a walk-off home run. A pitcher throws a no-hitter. A team gets red hot going into the Postseason. We know some of the catalysts that increase baseball fan interest. Now Major League Baseball (MLB) and Google Cloud want the Kaggle community’s help to identify the many other factors which pique supporter engagement and create deeper relationships betweens players and fans.\n\n\n\n
Problem statement
\n\n\nIn this competition, you’ll predict how fans engage with MLB players’ digital content on a daily basis for a future date range. You’ll have access to player performance data, social media data, and team factors like market size. Successful models will provide new insights into what signals most strongly correlate with and influence engagement.","metadata":{}},{"cell_type":"code","source":"import numpy as np \nimport pandas as pd \nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport warnings\nwarnings.filterwarnings('ignore')","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2021-06-28T18:44:03.968220Z","iopub.execute_input":"2021-06-28T18:44:03.968593Z","iopub.status.idle":"2021-06-28T18:44:04.039121Z","shell.execute_reply.started":"2021-06-28T18:44:03.968561Z","shell.execute_reply":"2021-06-28T18:44:04.038304Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = pd.read_csv(r'../input/mlb-player-digital-engagement-forecasting/train.csv')\nteams = pd.read_csv(r'../input/mlb-player-digital-engagement-forecasting/teams.csv')\nseasons = pd.read_csv(r'../input/mlb-player-digital-engagement-forecasting/seasons.csv')\nplayers = pd.read_csv(r'../input/mlb-player-digital-engagement-forecasting/players.csv')\nexample_test = pd.read_csv(r'../input/mlb-player-digital-engagement-forecasting/example_test.csv')\nsample_submission = pd.read_csv(r'../input/mlb-player-digital-engagement-forecasting/example_sample_submission.csv')\nawards = pd.read_csv(r'../input/mlb-player-digital-engagement-forecasting/awards.csv')","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2021-06-28T18:44:08.466229Z","iopub.execute_input":"2021-06-28T18:44:08.466578Z","iopub.status.idle":"2021-06-28T18:45:18.543021Z","shell.execute_reply.started":"2021-06-28T18:44:08.466546Z","shell.execute_reply":"2021-06-28T18:45:18.542087Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\n
train & test
\n\n\nRefrence: [https://www.kaggle.com/chumajin/eda-of-mlb-for-starter-english-ver](https://www.kaggle.com/chumajin/eda-of-mlb-for-starter-english-ver)\n\ndo check him out!","metadata":{}},{"cell_type":"code","source":"example_test.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-16T13:55:41.956988Z","iopub.execute_input":"2021-06-16T13:55:41.957336Z","iopub.status.idle":"2021-06-16T13:55:42.125652Z","shell.execute_reply.started":"2021-06-16T13:55:41.957302Z","shell.execute_reply":"2021-06-16T13:55:42.124712Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It seems that various information is included in json format.\n\nThe test data consists of one line of data per day.\n\n\nFor example, using a function from the starter code, it can be expanded as follows.","metadata":{}},{"cell_type":"code","source":"# Helper function to unpack json found in daily data\ndef unpack_json(json_str):\n return np.nan if pd.isna(json_str) else pd.read_json(json_str)","metadata":{"execution":{"iopub.status.busy":"2021-06-16T13:58:51.51322Z","iopub.execute_input":"2021-06-16T13:58:51.513543Z","iopub.status.idle":"2021-06-16T13:58:51.517748Z","shell.execute_reply.started":"2021-06-16T13:58:51.513515Z","shell.execute_reply":"2021-06-16T13:58:51.516607Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Visualize the example_test[\"games\"].iloc[0]","metadata":{}},{"cell_type":"code","source":"unpack_json(example_test[\"games\"].iloc[0])","metadata":{"execution":{"iopub.status.busy":"2021-06-16T14:07:05.563231Z","iopub.execute_input":"2021-06-16T14:07:05.563572Z","iopub.status.idle":"2021-06-16T14:07:05.609292Z","shell.execute_reply.started":"2021-06-16T14:07:05.563542Z","shell.execute_reply":"2021-06-16T14:07:05.60852Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Visualize the example_test[\"rosters\"].iloc[0]","metadata":{}},{"cell_type":"code","source":"unpack_json(example_test[\"rosters\"].iloc[0])","metadata":{"execution":{"iopub.status.busy":"2021-06-15T14:31:24.64576Z","iopub.execute_input":"2021-06-15T14:31:24.646473Z","iopub.status.idle":"2021-06-15T14:31:24.67987Z","shell.execute_reply.started":"2021-06-15T14:31:24.646434Z","shell.execute_reply":"2021-06-15T14:31:24.67848Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" Based on the above, we will look at other data such as train data","metadata":{}},{"cell_type":"code","source":"train.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-15T14:32:16.646992Z","iopub.execute_input":"2021-06-15T14:32:16.647559Z","iopub.status.idle":"2021-06-15T14:32:16.690794Z","shell.execute_reply.started":"2021-06-15T14:32:16.647507Z","shell.execute_reply":"2021-06-15T14:32:16.689818Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#date is converted to datetime\ntrain['date'] = pd.to_datetime(train['date'], format=\"%Y%m%d\")","metadata":{"execution":{"iopub.status.busy":"2021-06-15T14:32:37.204166Z","iopub.execute_input":"2021-06-15T14:32:37.204534Z","iopub.status.idle":"2021-06-15T14:32:37.21716Z","shell.execute_reply.started":"2021-06-15T14:32:37.2045Z","shell.execute_reply":"2021-06-15T14:32:37.215895Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### From here Let's take a look at one json where there is data for each column.\nAs mentioned above, the contents of train.csv are also in json file format in one cell, and it has a complicated shape like dataframe is further contained.\n\n(According to the result, it is an image that about 11 DataFrames (although it is a little small even if there is nan) are hanging as the amount of information for one day of data for 1216 days in total, which is a considerable amount of information. is.\n\nSo it's a little long from here. You may just grab the image and read it through. )","metadata":{}},{"cell_type":"code","source":"train.columns","metadata":{"execution":{"iopub.status.busy":"2021-06-28T20:18:31.034047Z","iopub.execute_input":"2021-06-28T20:18:31.034419Z","iopub.status.idle":"2021-06-28T20:18:31.043649Z","shell.execute_reply.started":"2021-06-28T20:18:31.034387Z","shell.execute_reply":"2021-06-28T20:18:31.042654Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It is troublesome to input one by one, so I will omit NaN and make the nth (0 is the top) sample as a dataframe and create a function to see the column name and contents.","metadata":{}},{"cell_type":"code","source":"def exshow(col,n):\n tmp = train[col]\n tmp = tmp.dropna()\n tmpdf = unpack_json(tmp.iloc[n])\n print(tmpdf.columns)\n return tmpdf","metadata":{"execution":{"iopub.status.busy":"2021-06-15T14:35:38.896271Z","iopub.execute_input":"2021-06-15T14:35:38.896715Z","iopub.status.idle":"2021-06-15T14:35:38.903236Z","shell.execute_reply.started":"2021-06-15T14:35:38.896679Z","shell.execute_reply":"2021-06-15T14:35:38.901634Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\n
1.1 nextDayPlayerEngagement (column 1 of train.csv)
\n\nNested JSON containing all modeling targets from the following day.","metadata":{}},{"cell_type":"code","source":"tmpdf = exshow(\"nextDayPlayerEngagement\",0)\ntmpdf.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-15T14:54:09.315672Z","iopub.execute_input":"2021-06-15T14:54:09.316026Z","iopub.status.idle":"2021-06-15T14:54:09.355277Z","shell.execute_reply.started":"2021-06-15T14:54:09.315997Z","shell.execute_reply":"2021-06-15T14:54:09.354142Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* engagementMetricsDate - date of player engagement metrics, based on US Pacific Time (aligns with previous day’s games, rosters, on-field statistics, transactions, awards, etc.).\n* playerId\n* target1\n* target2\n* target3\n* target4\n\n\ntarget1-target4 are each daily indexes of digital engagement on a 0-100 scale.","metadata":{}},{"cell_type":"markdown","source":"From here, plyaerId and targets1 to 4 from the next day onwards are pulled out.","metadata":{}},{"cell_type":"markdown","source":"\n\n
1.2 games(column 2 of train.csv)
\n\nNested JSON containing all game information for a given day. Includes spring training and exhibition games along with regular season, Postseason, and All-Star games.","metadata":{}},{"cell_type":"code","source":"tmpdf = exshow(\"games\",1) \ntmpdf.head(3)","metadata":{"execution":{"iopub.status.busy":"2021-06-15T14:53:58.328576Z","iopub.execute_input":"2021-06-15T14:53:58.328997Z","iopub.status.idle":"2021-06-15T14:53:58.376144Z","shell.execute_reply.started":"2021-06-15T14:53:58.328966Z","shell.execute_reply":"2021-06-15T14:53:58.374873Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Please open ↓ for the meaning of the column. (Because it is long, I hide it.)","metadata":{}},{"cell_type":"markdown","source":"* gamePk - Unique identifier for a game.\n* gameType - Game type, various types can be found here.\n* season\n* gameDate\n* gameTimeUTC - First pitch time in UTC.\n* resumeDate - Time game was resumed (if abandoned, otherwise null).\n* resumedFrom - Time game was originally abandoned (if abandoned, otherwise null).\n* codedGameState - Game status code, various types can be found here.\n* detailedGameState - Game status, various types can be found here.\n* isTie - Boolean, true if game ended in a tie.\n* gameNumber - Game number flag to distinguish double headers - 1 or 2.\n* doubleHeader - Y for DH, N for single game, S for split-ticket.\n* dayNight - Day or night flag for scheduled start time.\n* scheduledInnings - Scheduled number of innings.\n* gamesInSeries - Number of games in current series.\n* seriesDescription - Text description of current series.\n* homeId - Unique identifier for home team.\n* homeName - Home team name.\n* homeAbbrev - Home team abbreviation.\n* homeWins - Number of current wins on season for home team.\n* homeLosses - Number of current losses on season for home team.\n* homeWinPct - Current winning percentage for home team.\n* homeWinner - Boolean, true if home team wins.\n* homeScore - Runs scored by home team.\n* awayId - Unique identifier for away team.\n* awayName - Unique identifier for away team.\n* awayAbbrev - Away team abbreviation.\n* awayWins - Number of current wins on season for away team.\n* awayLosses - Number of current losses on season for away team.\n* awayWinPct - Current winning percentage for away team.\n* awayWinner - Boolean, true if away team wins.\n* awayScore - Runs scored by away team.","metadata":{"_kg_hide-input":true}},{"cell_type":"markdown","source":" \n\n
1.3 rosters(column 3 of train.csv)
\n\nNested JSON that contains all the roster information for a particular day. Includes in-season and off-season team rosters.","metadata":{}},{"cell_type":"code","source":"tmpdf = exshow(\"rosters\",0) \ntmpdf.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-15T14:53:32.477062Z","iopub.execute_input":"2021-06-15T14:53:32.477424Z","iopub.status.idle":"2021-06-15T14:53:32.502739Z","shell.execute_reply.started":"2021-06-15T14:53:32.477394Z","shell.execute_reply":"2021-06-15T14:53:32.501626Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* playerId - Unique identifier for a player.\n* gameDate\n* teamId - teamId that player is on that date.\n* statusCode - Roster status abbreviation.\n* status - Descriptive roster status.","metadata":{}},{"cell_type":"markdown","source":"\n
1.4 playerBoxScores(column 4 of train.csv)
\n\nNested JSON containing game stats aggregated at the player game level for a given day. Includes regular","metadata":{}},{"cell_type":"code","source":"tmpdf = exshow(\"playerBoxScores\",0) \ntmpdf.head(3)","metadata":{"execution":{"iopub.status.busy":"2021-06-15T14:53:06.544857Z","iopub.execute_input":"2021-06-15T14:53:06.545367Z","iopub.status.idle":"2021-06-15T14:53:06.635249Z","shell.execute_reply.started":"2021-06-15T14:53:06.545317Z","shell.execute_reply":"2021-06-15T14:53:06.634045Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Please open ↓ for the meaning of the column. (Because it is long, I hide it.)","metadata":{}},{"cell_type":"markdown","source":"* home - Binary, 1 if home team, 0 if away.\n* gamePk - unique identifier for game.\n* gameDate\n* gameTimeUTC - First pitch time in UTC.\n* teamId - Unique identifier for team.\n* teamName\n* playerId - Unique identifier for player.\n* playerName\n* jerseyNum\n* positionCode - Number position code, details are here.\n* positionName - Text position display, details are here.\n* positionType - Position group, details are here.\n* battingOrder - Format: “###”, where the first digit indicates the batting order spot, and the second two digits indicate the sequence in which that player occupied that batting order spot. Examples: “300”, which indicates the starter in the third spot in the batting order; “903,” which indicates the fourth player (after 900, 901 and 902) occupy the ninth spot in the batting order. Only populates if appeared in game.\n* gamesPlayedBatting - 1 if player entered the game as a batter, runner or fielder.\n* flyOuts - Game total fly outs.\n* groundOuts - Game total ground outs.\n* runsScored - Game total runs scored.\n* doubles - Game total doubles.\n* triples - Game total triples.\n* homeRuns - Game total home runs.\n* strikeOuts - Game total strike outs.\n* baseOnBalls - Game total walks.\n* intentionalWalks - Game total intentional walks.\n* hits - Game total hits.\n* hitByPitch - Game total hit by pitches.\n* atBats - Game total at-bats.\n* caughtStealing - Game total caught stealing.\n* stolenBases - Game total stolen bases.\n* groundIntoDoublePlay - Game total double plays grounded into.\n* groundIntoTriplePlay - Game total triple plays grounded into.\n* plateAppearances - Game total plate appearances.\n* totalBases - Game total bases.\n* rbi - Game total runs batted in.\n* leftOnBase - Game total runners left on base.\n* sacBunts - Game total sacrifice bunts.\n* sacFlies - Game total sacrifice flies.\n* catchersInterference - Game total catchers interference reached on.\n* pickoffs - Game total number of times picked off base.\n* gamesPlayedPitching- Binary, 1 if player entered the game as a pitcher.\n* gamesStartedPitching- Binary, 1 if player was game’s starting pitcher.\n* completeGamesPitching - Binary, 1 if credited with complete game.\n* shutoutsPitching - Binary, 1 if credited with shutout.\n* winsPitching - Binary, 1 if credited with win.\n* lossesPitching - Binary, 1 if credited with loss.\n* flyOutsPitching - Game total of flyouts allowed.\n* airOutsPitching - Game total of air outs (flyouts + popouts) allowed.\n* groundOutsPitching - Game total ground outs allowed.\n* runsPitching - Game total runs allowed.\n* doublesPitching - Game total doubles allowed.\n* triplesPitching - Game total triples allowed.\n* homeRunsPitching - Game total home runs allowed.\n* strikeOutsPitching - Game total strike outs allowed.\n* baseOnBallsPitching - Game total walks allowed.\n* intentionalWalksPitching - Game total intentional walks allowed.\n* hitsPitching - Game total hits allowed.\n* hitByPitchPitching - Game total hit by pitches allowed.\n* atBatsPitching - Game total at-bats against.\n* caughtStealingPitching - Game total caught stealing against.\n* stolenBasesPitching - Game total stolen bases allowed.\n* inningsPitched - Game total innings pitched.\n* saveOpportunities - Binary, 1 if credited with save opportunity.\n* earnedRuns - Game total earned runs allowed.\n* battersFaced - Game total batters faced.\n* outsPitching - Game total outs recorded.\n* pitchesThrown - Game total number of pitches thrown.\n* balls - Game total balls thrown.\n* strikes - Game total strikes thrown.\n* hitBatsmen - Game total hit batters.\n* balks - Game total balks.\n* wildPitches - Game total number of wild pitches thrown.\n* pickoffsPitching - Game total number of pickoffs.\n* rbiPitching - Game total number of runs batted in allowed.\n* inheritedRunners - Game total of inherited runners assumed.\n* inheritedRunnersScored- Game total of inherited runners that scored.\n* catchersInterferencePitching - Game total of catcher’s interference occurred by battery.\n* sacBuntsPitching - Game total sacrifice bunts allowed.\n* sacFliesPitching - Game total sacrifice flies allowed.\n* saves - Binary, 1 if credited with save.\n* holds - Binary, 1 if credited with hold.\n* blownSaves - Binary, 1 if credited with blown save.\n* assists - Game total number of assists.\n* putOuts - Game total number of putouts.\n* errors - Game total number of errors.\n* chances - Game total fielding chances.","metadata":{"_kg_hide-input":true}},{"cell_type":"markdown","source":"\n\n
1.5 teamBoxScores(column 5 of train.csv)
\n\nNested JSON containing game stats aggregated at the team game level for a given day. Includes regular season, Postseason, and All-Star games.","metadata":{}},{"cell_type":"code","source":"tmpdf = exshow(\"teamBoxScores\",0) \ntmpdf.head(3)","metadata":{"execution":{"iopub.status.busy":"2021-06-15T14:52:42.975636Z","iopub.execute_input":"2021-06-15T14:52:42.976093Z","iopub.status.idle":"2021-06-15T14:52:43.030869Z","shell.execute_reply.started":"2021-06-15T14:52:42.976054Z","shell.execute_reply":"2021-06-15T14:52:43.029634Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Please open ↓ for the meaning of the column. (Because it is long, I hide it.)","metadata":{}},{"cell_type":"markdown","source":"* home - Binary, 1 if home team, 0 if away.\n* teamId - unique identifier for team.\n* gamePk - unique identifier for game.\n* gameDate\n* gameTimeUTC - First pitch time in UTC.\n* flyOuts - Game total fly outs.\n* groundOuts - Game total ground outs.\n* runsScored - Game total runs scored.\n* doubles - Game total doubles.\n* triples - Game total triples.\n* homeRuns - Game total home runs.\n* strikeOuts - Game total strike outs.\n* baseOnBalls - Game total walks.\n* intentionalWalks - Game total intentional walks.\n* hits - Game total hits.\n* hitByPitch - Game total hit by pitches.\n* atBats - Game total at-bats.\n* caughtStealing - Game total caught stealing.\n* stolenBases - Game total stolen bases.\n* groundIntoDoublePlay - Game total double plays grounded into.\n* groundIntoTriplePlay - Game total triple plays grounded into.\n* plateAppearances - Game total plate appearances.\n* totalBases - Game total bases.\n* rbi - Game total runs batted in.\n* leftOnBase - Game total runners left on base.\n* sacBunts - Game total sacrifice bunts.\n* sacFlies - Game total sacrifice flies.\n* catchersInterference - Game total catcher’s interference reached on.\n* pickoffs - Game total number of times picked off base.\n* airOutsPitching - Game total of air outs (flyouts + popouts) allowed.\n* groundOutsPitching - Game total ground outs allowed.\n* runsPitching - Game total runs allowed.\n* doublesPitching - Game total doubles allowed.\n* triplesPitching - Game total triples allowed.\n* homeRunsPitching - Game total home runs allowed.\n* strikeOutsPitching - Game total strike outs allowed.\n* baseOnBallsPitching - Game total walks allowed.\n* intentionalWalksPitching - Game total intentional walks allowed.\n* hitsPitching - Game total hits allowed.\n* hitByPitchPitching - Game total hit by pitches allowed.\n* atBatsPitching - Game total at-bats against.\n* caughtStealingPitching - Game total caught stealing against.\n* stolenBasesPitching - Game total stolen bases allowed.\n* inningsPitched - Game total innings pitched.\n* earnedRuns - Game total earned runs allowed.\n* battersFaced - Game total batters faced.\n* outsPitching - Game total outs recorded.\n* hitBatsmen - Game total hit batters.\n* balks - Game total balks.\n* wildPitches - Game total number of wild pitches thrown.\n* pickoffsPitching - Game total number of pickoffs.\n* rbiPitching - Game total number of runs batted in allowed.\n* inheritedRunners - Game total of inherited runners assumed.\n* inheritedRunnersScored- Game total of inherited runners that scored.\n* catchersInterferencePitching - Game total of catcher’s interference occurred by battery.\n* sacBuntsPitching - Game total sacrifice bunts allowed.\n* sacFliesPitching - Game total sacrifice flies allowed.","metadata":{"_kg_hide-input":true}},{"cell_type":"markdown","source":" \n\n
1.6 transactions(column 6 of train.csv)
\n\nNested JSON containing all transaction information involving MLB teams for a given day.","metadata":{}},{"cell_type":"code","source":"tmpdf = exshow(\"transactions\",1) \ntmpdf.head(3)","metadata":{"execution":{"iopub.status.busy":"2021-06-15T14:52:16.103717Z","iopub.execute_input":"2021-06-15T14:52:16.104156Z","iopub.status.idle":"2021-06-15T14:52:16.137116Z","shell.execute_reply.started":"2021-06-15T14:52:16.10412Z","shell.execute_reply":"2021-06-15T14:52:16.136183Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* transactionId - Unique identifier for a transaction.\n* playerId - Unique identifier for a player.\n* playerName\n* date\n* fromTeamId - Unique identifier for the team a player goes from.\n* fromTeamName\n* toTeamId - Unique identifier for the team a player goes to.\n* toTeamName\n* effectiveDate\n* resolutionDate\n* typeCode - Transaction status abbreviation.\n* typeDesc - Transaction status description.\n* description - Text description of transaction.","metadata":{}},{"cell_type":"markdown","source":"\n\n
1.7 standings(column 7 of train.csv)
\n\nNested JSON containing all standings information involving MLB teams for a given day.","metadata":{}},{"cell_type":"code","source":"tmpdf = exshow(\"standings\",0) \ntmpdf.head(3)","metadata":{"execution":{"iopub.status.busy":"2021-06-15T14:51:55.713618Z","iopub.execute_input":"2021-06-15T14:51:55.713969Z","iopub.status.idle":"2021-06-15T14:51:55.764447Z","shell.execute_reply.started":"2021-06-15T14:51:55.71394Z","shell.execute_reply":"2021-06-15T14:51:55.763221Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Please open ↓ for the meaning of the column. (Because it is long, I hide it.)","metadata":{}},{"cell_type":"markdown","source":"* season\n* gameDate\n* divisionId - Unique identifier of the division that this team plays in.\n* teamId - Unique identifier for a team.\n* teamName\n* streakCode - Abbreviation of team’s current win or loss streak. First letter indicates winning or losing, number is the number of games.\n* divisionRank - Team’s current rank in their division.\n* leagueRank - Team’s current rank in their league.\n* wildCardRank - Team’s current rank for a wildcard berth.\n* leagueGamesBack - Games back in team’s league.\n* sportGamesBack - Games back in all of MLB.\n* divisionGamesBack - Games back in team’s division.\n* wins - Current wins.\n* losses - Current losses.\n* pct - Current winning percentage.\n* runsAllowed - Runs allowed on the season.\n* runsScored - Runs scored on the season.\n* divisionChamp - true if team has clinched the division title.\n* divisionLeader - true if team is leading the division race.\n* wildCardLeader - true if team is wildcard leader.\n* eliminationNumber - number of games (team losses + opponent wins) before eliminated from the division race.\n* wildCardEliminationNumber - number of games (team losses + opponent wins) before eliminated from the wildcard race.\n* homeWins - Home wins on the season.\n* homeLosses - Home losses on the season.\n* awayWins - Away wins on the season.\n* awayLosses - Away losses on the season.\n* lastTenWins - Wins in the last 10 games.\n* lastTenLosses - Losses in the last 10 games.\n* extraInningWins - Wins in extra innings on the season.\n* extraInningLosses - Losses in extra innings on the season.\n* oneRunWins - Wins by one run on the season.\n* oneRunLosses - Losses by one run on the season.\n* dayWins - Day game wins on the season.\n* dayLosses Day game losses on the season.\n* nightWins - Night game wins on the season.\n* nightLosses - Night game losses on the season.\n* grassWins - Grass field wins on the season.\n* grassLosses - Grass field losses on the season.\n* turfWins - Turf field wins on the season.\n* turfLosses - Turf field losses on the season.\n* divWins - Wins against division opponents on the season.\n* divLosses - Losses against division opponents on the season.\n* alWins - Wins against AL teams on the season.\n* alLosses - Losses against AL teams on the season.\n* nlWins - Wins against NL teams on the season.\n* nlLosses - Losses against NL teams on the season.\n* xWinLossPct - Expected winning percentage based on runs scored and allowed.","metadata":{"_kg_hide-input":true}},{"cell_type":"markdown","source":"\n\n
1.8 awards(column 8 of train.csv)
\n\nNested JSON containing all awards or honors handed out on a given day.","metadata":{}},{"cell_type":"code","source":"tmpdf = exshow(\"awards\",0) \ntmpdf","metadata":{"execution":{"iopub.status.busy":"2021-06-15T14:51:34.124653Z","iopub.execute_input":"2021-06-15T14:51:34.125021Z","iopub.status.idle":"2021-06-15T14:51:34.148104Z","shell.execute_reply.started":"2021-06-15T14:51:34.124987Z","shell.execute_reply":"2021-06-15T14:51:34.146868Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* awardId\n* awardName\n* awardDate - Date award was given.\n* awardSeason - Season award was from.\n* playerId - Unique identifier for a player.\n* playerName\n* awardPlayerTeamId","metadata":{}},{"cell_type":"markdown","source":"\n\n
1.9 events(column 9 of train.csv)
\n\nNested JSON containing all on-field game events for a given day. Includes regular season and Postseason games.","metadata":{}},{"cell_type":"code","source":"tmpdf = exshow(\"events\",0) \ntmpdf.head(3)","metadata":{"execution":{"iopub.status.busy":"2021-06-15T14:51:26.085098Z","iopub.execute_input":"2021-06-15T14:51:26.085547Z","iopub.status.idle":"2021-06-15T14:51:26.407344Z","shell.execute_reply.started":"2021-06-15T14:51:26.085507Z","shell.execute_reply":"2021-06-15T14:51:26.406344Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Please open ↓ for the meaning of the column. (Because it is long, I hide it.)","metadata":{}},{"cell_type":"markdown","source":"* gamePk - unique identifier for game.\n* gameDate\n* gameTimeUTC - First pitch time in UTC.\n* season\n* gameType - Game type, various types can be found here.\n* playId - Statcast play guid.\n* eventId\n* inning - Inning AB occurred.\n* halfInning- “top” or “bottom” inning indicator.\n* homeScore - Home score at start of the event.\n* awayScore - Away score at start of the event.\n* menOnBase - Split used if runner on base – ie (RISP, Empty).\n* atBatIndex - At-Bat index number, begins with 0.\n* atBatDesc - Play-by-play description of AB.\n* atBatEvent - Event type result of the atBat, various types can be found here.\n* hasOut - Binary, 1 if runner is out on play.\n* pitcherTeamId - Unique identifier for pitching team.\n* isPitcherHome - Binary, 1 if pitcher is home team.\n* pitcherTeam - team name of pitching team.\n* hitterTeamId - Unique identifier for hitting team.\n* hitterTeam - Team name of hitting team.\n* pitcherId\n* pitcherName\n* isStarter - Binary, 1 if player was game’s starting pitcher.\n* pitcherHand - Player throw hand: “L”,”R”.\n* hitterId\n* hitterName\n* batSide - Player bat side: “L”,”R”.\n* pitchNumber - Pitch sequence number of AB.\n* balls - Ball count after the event.\n* strikes - Strike count after the event.\n* isGB - Binary, 1 if at bat is a ground ball.\n* isLD - Binary, 1 if at bat is a line drive.\n* isFB - Binary, 1 if at bat is a fly ball.\n* isPU - Binary, 1 if at bat is a pop up.\n* launchSpeed - Measured speed of the hit ball.\n* launchAngle - Vertical angle relative to the horizon at which the hit was launched.\n* totalDistance - Total distance ball traveled.\n* event - Events that may occur during an at-bat: strikeout, groundout, substitutions, stolen base, wild pitch, game advisory, etc.\n* description - Text description of event.\n* rbi - Number of runs batted in during AB.\n* pitchType - Pitch type classification code, various types can be found here.\n* call - Pitch or throw result classification code, various types can be found here.\n* outs - Current/Final outs of AB.\n* inPlay - True/False if ball was hit into play.\n* isPaOver - Binary, 1 if this event is the end of the plate appearance.\n* startSpeed - Speed in MPH of the ball at 50 feet in front of home plate.\n* endSpeed - Speed in MPH of the ball as it crosses the front edge of home plate (0,0 in the x axis).\n* nastyFactor - Evaluates several properties of each pitch, and rates the \"nastiness\" of the pitch on a scale from 0-100.\n* breakAngle - Degrees clockwise (batter’s view) that the plane of the pitch deviates from the vertical.\n* breakLength - Max distance that the pitch separates from the straight line between pitch start and pitch end.\n* breakY - Distance from home plate where the break is greatest.\n* spinRate - Rate of spin on the ball after it was released by pitcher in RPMs.\n* spinDirection - The axis of rotation for the ball at release given as an angle that reflects how the spin will influence the ball trajectory. Pure back-spin is 180 degrees, pure side-spin that puts the ball to the 1b side is 90 degrees, pure-side spin that pulls the ball to the 3b side is 270 degrees, and pure top-spin is 0 or 360 degrees.\n* pX - Horizontal position in feet of the ball as it crosses the front axis of home plate.\n* pZ - Vertical position in feet above home plate of the ball as it crosses the front axis of home plate.\n* aX - Ball acceleration on the z axis.\n* aY - Ball acceleration on the y axis.\n* aZ - Ball acceleration on the z axis.\n* pfxX - Horizontal movement of the ball in inches.\n* pfxZ - Vertical movement of the ball in inches.\n* vX0 - Velocity of the ball from the x axis.\n* vY0 - Velocity of the ball from the y axis; this is negative because 0,0,0 is behind the batter and the ball travels from pitcher mound towards 0,0,0.\n* vZ0 - Velocity of the ball from the z axis.\n* x - X coordinate where pitch crossed front of home plate.\n* y - Y coordinate where pitch crossed front of home plate.\n* x0 - Coordinate location of the ball at the point it was released from the pitchers hand on the x axis (time = 0).\n* y0 - Coordinate location of the ball at the point it was released from the pitchers hand on the y axis (time = 0).\n* z0 - Coordinate location of the ball at the point it was released from the pitchers hand on the z axis (time = 0).\n* type - Type of event, either ‘pitch’ or ‘action’\n* zone - Zone location number - see image below.","metadata":{"_kg_hide-input":true}},{"cell_type":"markdown","source":"\n
1.10 playerTwitterFollowers(column 10 of train.csv)
\n\nNested JSON containing some players’ number of Twitter followers on that day.","metadata":{}},{"cell_type":"code","source":"tmpdf = exshow(\"playerTwitterFollowers\",0) \ntmpdf.head(3)","metadata":{"execution":{"iopub.status.busy":"2021-06-15T14:49:46.942157Z","iopub.execute_input":"2021-06-15T14:49:46.942637Z","iopub.status.idle":"2021-06-15T14:49:46.987126Z","shell.execute_reply.started":"2021-06-15T14:49:46.942577Z","shell.execute_reply":"2021-06-15T14:49:46.98597Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Twitter following data was collected by MLB from Twitter APIs for Major League players, on the first of each month, dating back to 1/1/2018. This dataset is not exhaustive of all players over all months, as not every player has/had a Twitter account, players may create/delete/reinstate accounts at random, or other scenarios preventing follower data from being collected on a given date.\n\n* date - Date of follower count.\n* playerId - Unique identifier for a player.\n* playerName\n* accountName - Name on player twitter account.\n* twitterHandle - Player twitter handle.\n* numberOfFollowers - Number of followers","metadata":{}},{"cell_type":"markdown","source":"\n\n
1.11 teamTwitterFollowers(column 11 of train.csv)
\n\n Nested JSON containing each team’s number of Twitter followers on that day.","metadata":{}},{"cell_type":"code","source":"tmpdf = exshow(\"teamTwitterFollowers\",0) \ntmpdf.head(3)","metadata":{"execution":{"iopub.status.busy":"2021-06-15T14:51:07.886275Z","iopub.execute_input":"2021-06-15T14:51:07.886763Z","iopub.status.idle":"2021-06-15T14:51:07.914238Z","shell.execute_reply.started":"2021-06-15T14:51:07.886719Z","shell.execute_reply":"2021-06-15T14:51:07.912871Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Twitter following data was collected by MLB from Twitter APIs for all 30 Major League teams, on the first of each month, dating back to 1/1/2018.\n\n* date - Date of follower count.\n* teamId - Unique identifier for a team.\n* teamName\n* accountName - Name on team twitter account.\n* twitterHandle - Team twitter handle.\n* numberOfFollowers - Number of followers.","metadata":{}},{"cell_type":"markdown","source":"\n\n
teams.csv
\n\n\nLibrary containing high level information about all MLB teams.\n","metadata":{}},{"cell_type":"code","source":"teams.head()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-06-15T14:56:04.238128Z","iopub.execute_input":"2021-06-15T14:56:04.238547Z","iopub.status.idle":"2021-06-15T14:56:04.259519Z","shell.execute_reply.started":"2021-06-15T14:56:04.238511Z","shell.execute_reply":"2021-06-15T14:56:04.258225Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('teams Columns:')\nteams.columns","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-06-15T14:56:09.016972Z","iopub.execute_input":"2021-06-15T14:56:09.017421Z","iopub.status.idle":"2021-06-15T14:56:09.02684Z","shell.execute_reply.started":"2021-06-15T14:56:09.017377Z","shell.execute_reply":"2021-06-15T14:56:09.025456Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import plotly.express as px\n\nfig = px.bar(teams.columns[:-1] , x=teams.nunique(), y=teams.columns, orientation='h')\n\nfig.update_layout(title='Feature cardinality in teams set',\n xaxis_title='unique values',\n yaxis_title='features',\n titlefont={'size': 28, 'family':'Serif'},\n template='simple_white',\n showlegend=True,\n width=900, height=1000)\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-06-15T14:56:10.795974Z","iopub.execute_input":"2021-06-15T14:56:10.796378Z","iopub.status.idle":"2021-06-15T14:56:14.039712Z","shell.execute_reply.started":"2021-06-15T14:56:10.796344Z","shell.execute_reply":"2021-06-15T14:56:14.038641Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> ### \n\n
Players.csv
\n\n\nLibrary containing high level information about all MLB players in this dataset.","metadata":{}},{"cell_type":"code","source":"players.head()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-06-15T14:56:43.646815Z","iopub.execute_input":"2021-06-15T14:56:43.647267Z","iopub.status.idle":"2021-06-15T14:56:43.667139Z","shell.execute_reply.started":"2021-06-15T14:56:43.647231Z","shell.execute_reply":"2021-06-15T14:56:43.666068Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Players Columns:')\nplayers.columns","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-06-15T14:56:45.026306Z","iopub.execute_input":"2021-06-15T14:56:45.026949Z","iopub.status.idle":"2021-06-15T14:56:45.036242Z","shell.execute_reply.started":"2021-06-15T14:56:45.026854Z","shell.execute_reply":"2021-06-15T14:56:45.034759Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = px.bar(players.columns[:-1] , x=players.nunique(), y=players.columns, orientation='h')\n\nfig.update_layout(title='Feature cardinality in players set',\n xaxis_title='unique values',\n yaxis_title='features',\n titlefont={'size': 28, 'family':'Serif'},\n template='simple_white',\n showlegend=True,\n width=900, height=1000)\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-06-15T14:56:48.414227Z","iopub.execute_input":"2021-06-15T14:56:48.41457Z","iopub.status.idle":"2021-06-15T14:56:48.516091Z","shell.execute_reply.started":"2021-06-15T14:56:48.414541Z","shell.execute_reply":"2021-06-15T14:56:48.514584Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import missingno as msno\nfig1 = msno.bar(players)\nfont = {'family': 'serif',\n 'color': 'darkred',\n 'weight': 'normal',\n 'size': 30\n }\nplt.title(label='Bar chart representation of the null values',fontdict=font)\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-06-15T14:56:55.479878Z","iopub.execute_input":"2021-06-15T14:56:55.480336Z","iopub.status.idle":"2021-06-15T14:56:56.620309Z","shell.execute_reply.started":"2021-06-15T14:56:55.480301Z","shell.execute_reply":"2021-06-15T14:56:56.619262Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"msno.matrix(players)\nfont = {'family': 'serif',\n 'color': 'darkred',\n 'weight': 'normal',\n 'size': 30\n }\nplt.title(label='Null values matrix of the dataset',fontdict=font)\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-06-15T14:57:10.226145Z","iopub.execute_input":"2021-06-15T14:57:10.226885Z","iopub.status.idle":"2021-06-15T14:57:10.790576Z","shell.execute_reply.started":"2021-06-15T14:57:10.226833Z","shell.execute_reply":"2021-06-15T14:57:10.789549Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\n
Awards.csv
\n\n\nThis file has awards won by players in the training set prior to the beginning of the daily data (i.e. before 2018).\n\n* awardDate - Date award was given.\n* awardSeason - Season award was from.\n* awardId\n* awardName\n* playerId - Unique identifier for a player.\n* playerName\n* awardPlayerTeamId","metadata":{}},{"cell_type":"code","source":"awards.head()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-06-15T14:57:25.747176Z","iopub.execute_input":"2021-06-15T14:57:25.747575Z","iopub.status.idle":"2021-06-15T14:57:25.762155Z","shell.execute_reply.started":"2021-06-15T14:57:25.747543Z","shell.execute_reply":"2021-06-15T14:57:25.761321Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = px.bar(awards.columns[:-1] , x=awards.nunique(), y=awards.columns, orientation='h')\n\nfig.update_layout(title='Feature cardinality in awards set',\n xaxis_title='unique values',\n yaxis_title='features',\n titlefont={'size': 28, 'family':'Serif'},\n template='simple_white',\n showlegend=True,\n width=900, height=1000)\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-06-15T14:57:26.177815Z","iopub.execute_input":"2021-06-15T14:57:26.178433Z","iopub.status.idle":"2021-06-15T14:57:26.292094Z","shell.execute_reply.started":"2021-06-15T14:57:26.178374Z","shell.execute_reply":"2021-06-15T14:57:26.290946Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\n
Data merger
\n\n\nRefrence: [https://www.kaggle.com/columbia2131/mlb-lightgbm-starter-dataset-code-en-ja](https://www.kaggle.com/columbia2131/mlb-lightgbm-starter-dataset-code-en-ja)","metadata":{}},{"cell_type":"code","source":"from pathlib import Path\nTRAIN_DIR = Path('../input/mlb-pdef-train-dataset')\nrosters = pd.read_pickle(TRAIN_DIR / 'rosters_train.pkl')\ntargets = pd.read_pickle(TRAIN_DIR / 'nextDayPlayerEngagement_train.pkl')\nscores = pd.read_pickle(TRAIN_DIR / 'playerBoxScores_train.pkl')","metadata":{"execution":{"iopub.status.busy":"2021-06-28T18:45:31.385310Z","iopub.execute_input":"2021-06-28T18:45:31.385674Z","iopub.status.idle":"2021-06-28T18:45:34.998232Z","shell.execute_reply.started":"2021-06-28T18:45:31.385638Z","shell.execute_reply":"2021-06-28T18:45:34.997246Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"targets_cols = ['playerId', 'target1', 'target2', 'target3', 'target4', 'date']\nplayers_cols = ['playerId', 'primaryPositionName']\nrosters_cols = ['playerId', 'teamId', 'status', 'date']\nscores_cols = ['playerId', 'battingOrder', 'gamesPlayedBatting', 'flyOuts',\n 'groundOuts', 'runsScored', 'doubles', 'triples', 'homeRuns',\n 'strikeOuts', 'baseOnBalls', 'intentionalWalks', 'hits', 'hitByPitch',\n 'atBats', 'caughtStealing', 'stolenBases', 'groundIntoDoublePlay',\n 'groundIntoTriplePlay', 'plateAppearances', 'totalBases', 'rbi',\n 'leftOnBase', 'sacBunts', 'sacFlies', 'catchersInterference',\n 'pickoffs', 'gamesPlayedPitching', 'gamesStartedPitching',\n 'completeGamesPitching', 'shutoutsPitching', 'winsPitching','lossesPitching', 'flyOutsPitching', 'airOutsPitching',\n 'groundOutsPitching', 'runsPitching', 'doublesPitching',\n 'triplesPitching', 'homeRunsPitching', 'strikeOutsPitching',\n 'baseOnBallsPitching', 'intentionalWalksPitching', 'hitsPitching',\n 'hitByPitchPitching', 'atBatsPitching', 'caughtStealingPitching',\n 'stolenBasesPitching', 'inningsPitched', 'saveOpportunities',\n 'earnedRuns', 'battersFaced', 'outsPitching', 'pitchesThrown', 'balls',\n 'strikes', 'hitBatsmen', 'balks', 'wildPitches', 'pickoffsPitching',\n 'rbiPitching', 'gamesFinishedPitching', 'inheritedRunners',\n 'inheritedRunnersScored', 'catchersInterferencePitching',\n 'sacBuntsPitching', 'sacFliesPitching', 'saves', 'holds', 'blownSaves',\n 'assists', 'putOuts', 'errors', 'chances', 'date']\n\nfeature_cols = ['label_playerId', 'label_primaryPositionName', 'label_teamId',\n 'label_status', 'battingOrder', 'gamesPlayedBatting', 'flyOuts',\n 'groundOuts', 'runsScored', 'doubles', 'triples', 'homeRuns',\n 'strikeOuts', 'baseOnBalls', 'intentionalWalks', 'hits', 'hitByPitch',\n 'atBats', 'caughtStealing', 'stolenBases', 'groundIntoDoublePlay',\n 'groundIntoTriplePlay', 'plateAppearances', 'totalBases', 'rbi',\n 'leftOnBase', 'sacBunts', 'sacFlies', 'catchersInterference',\n 'pickoffs', 'gamesPlayedPitching', 'gamesStartedPitching',\n 'completeGamesPitching', 'shutoutsPitching', 'winsPitching',\n 'lossesPitching', 'flyOutsPitching', 'airOutsPitching',\n 'groundOutsPitching', 'runsPitching', 'doublesPitching',\n 'triplesPitching', 'homeRunsPitching', 'strikeOutsPitching','baseOnBallsPitching', 'intentionalWalksPitching', 'hitsPitching',\n 'hitByPitchPitching', 'atBatsPitching', 'caughtStealingPitching',\n 'stolenBasesPitching', 'inningsPitched', 'saveOpportunities',\n 'earnedRuns', 'battersFaced', 'outsPitching', 'pitchesThrown', 'balls',\n 'strikes', 'hitBatsmen', 'balks', 'wildPitches', 'pickoffsPitching',\n 'rbiPitching', 'gamesFinishedPitching', 'inheritedRunners',\n 'inheritedRunnersScored', 'catchersInterferencePitching',\n 'sacBuntsPitching', 'sacFliesPitching', 'saves', 'holds', 'blownSaves',\n 'assists', 'putOuts', 'errors', 'chances']","metadata":{"execution":{"iopub.status.busy":"2021-06-28T18:45:37.222621Z","iopub.execute_input":"2021-06-28T18:45:37.222999Z","iopub.status.idle":"2021-06-28T18:45:37.234979Z","shell.execute_reply.started":"2021-06-28T18:45:37.222966Z","shell.execute_reply":"2021-06-28T18:45:37.233817Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = targets[targets_cols].merge(players[players_cols], on=['playerId'], how='left')\ntrain = train.merge(rosters[rosters_cols], on=['playerId', 'date'], how='left')\ntrain = train.merge(scores[scores_cols], on=['playerId', 'date'], how='left')\n\n# label encoding\nplayer2num = {c: i for i, c in enumerate(train['playerId'].unique())}\nposition2num = {c: i for i, c in enumerate(train['primaryPositionName'].unique())}\nteamid2num = {c: i for i, c in enumerate(train['teamId'].unique())}\nstatus2num = {c: i for i, c in enumerate(train['status'].unique())}\ntrain['label_playerId'] = train['playerId'].map(player2num)\ntrain['label_primaryPositionName'] = train['primaryPositionName'].map(position2num)\ntrain['label_teamId'] = train['teamId'].map(teamid2num)\ntrain['label_status'] = train['status'].map(status2num)","metadata":{"execution":{"iopub.status.busy":"2021-06-28T18:45:39.907870Z","iopub.execute_input":"2021-06-28T18:45:39.908219Z","iopub.status.idle":"2021-06-28T18:45:43.734940Z","shell.execute_reply.started":"2021-06-28T18:45:39.908188Z","shell.execute_reply":"2021-06-28T18:45:43.733999Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
Building Model
\n","metadata":{}},{"cell_type":"code","source":"X = train[feature_cols]\ny = train[['target1', 'target2', 'target3', 'target4']]","metadata":{"execution":{"iopub.status.busy":"2021-06-28T18:45:45.826399Z","iopub.execute_input":"2021-06-28T18:45:45.826731Z","iopub.status.idle":"2021-06-28T18:45:47.336955Z","shell.execute_reply.started":"2021-06-28T18:45:45.826699Z","shell.execute_reply":"2021-06-28T18:45:47.336047Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"_index = (train['date'] < 20210401)\nx_train = X.loc[_index].reset_index(drop=True)\ny_train = y.loc[_index].reset_index(drop=True)\nx_valid = X.loc[~_index].reset_index(drop=True)\ny_valid = y.loc[~_index].reset_index(drop=True)","metadata":{"execution":{"iopub.status.busy":"2021-06-28T18:47:35.773236Z","iopub.execute_input":"2021-06-28T18:47:35.773573Z","iopub.status.idle":"2021-06-28T18:47:38.063091Z","shell.execute_reply.started":"2021-06-28T18:47:35.773543Z","shell.execute_reply":"2021-06-28T18:47:38.062099Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.metrics import mean_absolute_error\nfrom lightgbm import LGBMRegressor\n\ndef fit_lgbm(x_train, y_train, x_valid, y_valid, params: dict=None):\n oof_pred = np.zeros(len(y_valid), dtype=np.float32)\n model = LGBMRegressor(**params)\n model.fit(x_train, y_train, \n eval_set=[(x_valid, y_valid)], \n early_stopping_rounds=100, \n verbose=400)\n oof_pred = model.predict(x_valid)\n score = mean_absolute_error(oof_pred, y_valid)\n print('mae:', score)\n return oof_pred, model, score\n\n\n# training lightgbm\nparams = {\n 'objective':'mae',\n 'reg_alpha': 2.361501631985262,\n 'reg_lambda': 2.4000071578349456, \n 'n_estimators': 100000,\n 'learning_rate': 0.0011097822790813784,\n 'random_state': 42,\n}\n\noof1, model1, score1 = fit_lgbm(x_train, y_train['target1'],x_valid, y_valid['target1'],params)\noof2, model2, score2 = fit_lgbm(x_train, y_train['target2'],x_valid, y_valid['target2'],params)\noof3, model3, score3 = fit_lgbm(x_train, y_train['target3'],x_valid, y_valid['target3'],params)\noof4, model4, score4 = fit_lgbm(x_train, y_train['target4'],x_valid, y_valid['target4'],params)\n\nscore = (score1+score2+score3+score4) / 4\nprint(f'score: {score}')\n","metadata":{"execution":{"iopub.status.busy":"2021-06-28T18:53:25.304835Z","iopub.execute_input":"2021-06-28T18:53:25.305176Z","iopub.status.idle":"2021-06-28T20:13:40.861993Z","shell.execute_reply.started":"2021-06-28T18:53:25.305147Z","shell.execute_reply":"2021-06-28T20:13:40.861149Z"},"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\nplayers_cols = ['playerId', 'primaryPositionName']\nrosters_cols = ['playerId', 'teamId', 'status']\nscores_cols = ['playerId', 'battingOrder', 'gamesPlayedBatting', 'flyOuts',\n 'groundOuts', 'runsScored', 'doubles', 'triples', 'homeRuns',\n 'strikeOuts', 'baseOnBalls', 'intentionalWalks', 'hits', 'hitByPitch',\n 'atBats', 'caughtStealing', 'stolenBases', 'groundIntoDoublePlay',\n 'groundIntoTriplePlay', 'plateAppearances', 'totalBases', 'rbi',\n 'leftOnBase', 'sacBunts', 'sacFlies', 'catchersInterference',\n 'pickoffs', 'gamesPlayedPitching', 'gamesStartedPitching',\n 'completeGamesPitching', 'shutoutsPitching', 'winsPitching',\n 'lossesPitching', 'flyOutsPitching', 'airOutsPitching',\n 'groundOutsPitching', 'runsPitching', 'doublesPitching',\n 'triplesPitching', 'homeRunsPitching', 'strikeOutsPitching',\n 'baseOnBallsPitching', 'intentionalWalksPitching', 'hitsPitching',\n 'hitByPitchPitching', 'atBatsPitching', 'caughtStealingPitching',\n 'stolenBasesPitching', 'inningsPitched', 'saveOpportunities',\n 'earnedRuns', 'battersFaced', 'outsPitching', 'pitchesThrown', 'balls',\n 'strikes', 'hitBatsmen', 'balks', 'wildPitches', 'pickoffsPitching',\n 'rbiPitching', 'gamesFinishedPitching', 'inheritedRunners',\n 'inheritedRunnersScored', 'catchersInterferencePitching',\n 'sacBuntsPitching', 'sacFliesPitching', 'saves', 'holds', 'blownSaves',\n 'assists', 'putOuts', 'errors', 'chances']\n\nnull = np.nan\ntrue = True\nfalse = False\n\nimport mlb\n\nenv = mlb.make_env() # initialize the environment\niter_test = env.iter_test() # iterator which loops over each date in test set\n\nfor (test_df, sample_prediction_df) in iter_test: # make predictions here\n \n sample_prediction_df = sample_prediction_df.reset_index(drop=True)\n \n # creat dataset\n sample_prediction_df['playerId'] = sample_prediction_df['date_playerId']\\\n .map(lambda x: int(x.split('_')[1]))\n # Dealing with missing values\n if test_df['rosters'].iloc[0] == test_df['rosters'].iloc[0]:\n test_rosters = pd.DataFrame(eval(test_df['rosters'].iloc[0]))\n else:\n test_rosters = pd.DataFrame({'playerId': sample_prediction_df['playerId']})\n for col in rosters.columns:\n if col == 'playerId': continue\n test_rosters[col] = np.nan\n \n if test_df['playerBoxScores'].iloc[0] == test_df['playerBoxScores'].iloc[0]:\n test_scores = pd.DataFrame(eval(test_df['playerBoxScores'].iloc[0]))\n else:\n test_scores = pd.DataFrame({'playerId': sample_prediction_df['playerId']})\n for col in scores.columns:\n if col == 'playerId': continue\n test_scores[col] = np.nan\n test_scores = test_scores.groupby('playerId').sum().reset_index()\n test = sample_prediction_df[['playerId']].copy()\n test = test.merge(players[players_cols], on='playerId', how='left')\n test = test.merge(test_rosters[rosters_cols], on='playerId', how='left')\n test = test.merge(test_scores[scores_cols], on='playerId', how='left')\n\n test['label_playerId'] = test['playerId'].map(player2num)\n test['label_primaryPositionName'] = test['primaryPositionName'].map(position2num)\n test['label_teamId'] = test['teamId'].map(teamid2num)\n test['label_status'] = test['status'].map(status2num)\n \n test_X = test[feature_cols]\n \n # predict\n pred1 = model1.predict(test_X)\n pred2 = model2.predict(test_X)\n pred3 = model3.predict(test_X)\n pred4 = model4.predict(test_X)\n \n # merge submission\n sample_prediction_df['target1'] = np.clip(pred1, 0, 100)\n sample_prediction_df['target2'] = np.clip(pred2, 0, 100)\n sample_prediction_df['target3'] = np.clip(pred3, 0, 100)\n sample_prediction_df['target4'] = np.clip(pred4, 0, 100)\n sample_prediction_df = sample_prediction_df.fillna(0.)\n del sample_prediction_df['playerId']\n \n env.predict(sample_prediction_df)","metadata":{"execution":{"iopub.status.busy":"2021-06-28T20:15:47.522980Z","iopub.execute_input":"2021-06-28T20:15:47.523349Z","iopub.status.idle":"2021-06-28T20:15:59.085489Z","shell.execute_reply.started":"2021-06-28T20:15:47.523319Z","shell.execute_reply":"2021-06-28T20:15:59.084540Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
If you like this notebook, please upvote it! \n Thank you! :)
\n","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}