{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Back to the Kaggle\n<hr style=\"border:0.02in solid gray\"> </hr>\n\nThis is a visual exploratory data analysis for the MLB Player Digital Engagement Forecasting Competition. In fact, it is a kind of template I'm doing my self to restart my kaggle activity, so I thought, It will be better instead of have it private, to share some of the formats, code snippets and apply them to an actual active competition. So, I hope you find useful this notebook, not only for predictive modeling activity but also for code organization, documentation, formatting, data visualization and actually for other projects outside Kaggle.","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"# A Short Series: Exploratory-Data-Analysis\n<hr style=\"border:0.02in solid gray\"> </hr>\n\nMy intention is to produce notebooks iteratively until I feel comfortable to jump into a competition mode, so, here are a simple roadmap of intended Kaggle notebooks:\n\n1. Players Visual EDA (Short Version)\n2. Teams Visual EDA (Short Version)\n3. ... evaluate next content according to evidence of interaction and modeling findings (mainly notebooks or mainly competition)\n\nThere are two main reasons on why I'm sharing this:\n1. To draw your attention into the \"Documentation\" kind of aspect in these notebooks, so they help *you* win the competition.\n2. To ask for your comments, suggestions and hopefully a couple of votes if you found some content here useful for your particular purposes.","metadata":{}},{"cell_type":"code","source":"# -------------------------------------------------------------------------------------------- #\n# -- Kaggle Python 3 environment\n# -- Image://github.com/kaggle/docker-python\n# -------------------------------------------------------------------------------------------- #\n\n# -- Generic packages\nimport numpy as np   # Linear Algebra and Scientific Computing\nimport pandas as pd  # Data I/O and Processing\n\n# -- Operative System Navigation\nimport os            \nfrom os import listdir, path\nfrom os.path import isfile, join\n\n# -- Data Visualization\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport pandas_profiling as pp","metadata":{"execution":{"iopub.status.busy":"2021-06-10T23:54:18.935416Z","iopub.execute_input":"2021-06-10T23:54:18.935754Z","iopub.status.idle":"2021-06-10T23:54:19.897315Z","shell.execute_reply.started":"2021-06-10T23:54:18.935654Z","shell.execute_reply":"2021-06-10T23:54:19.896269Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# About the competition\n<hr style=\"border:0.02in solid gray\"> </hr>\n\nI found very useful and self-explanatory the first paragraph of the description tab in the competition, from that and for the sake of producing a quick and short version of Visual EDA, consider the following: \n\n- Time Series clasification problem (Multiclass): target1, target2, target3, target4.\n- You have info from other years but just forecast for players active in the 2021 season.\n- There are files with panel data (that do not change over time): players.csv, teams.csv, seasons.csv, awards.csv.\n- There is a file with time series data (daily data, train.csv).\n- Predicting the target in $d$ (in dataset), means to forecast target value for $d+1$ (in the problem context).","metadata":{}},{"cell_type":"markdown","source":"# Input data\n<hr style=\"border:0.02in solid gray\"> </hr>\n\nInput data files are available in the read-only \"../input/\" directory. Which contains:\n- A set of static files that do not change with time: players, seasons, teams, awards\n- A training dataset with daily labels (shifted +1 in time)","metadata":{}},{"cell_type":"code","source":"# Get the absolute path of the folder with train files\nfiles_path = path.abspath('/kaggle/input/mlb-player-digital-engagement-forecasting')\ndata_players = pd.read_csv(files_path + '/players.csv')\ndata_teams = pd.read_csv(files_path + '/teams.csv')\ndata_seasons = pd.read_csv(files_path + '/seasons.csv')\ndata_awards = pd.read_csv(files_path + '/awards.csv')","metadata":{"execution":{"iopub.status.busy":"2021-06-10T23:54:19.899051Z","iopub.execute_input":"2021-06-10T23:54:19.899353Z","iopub.status.idle":"2021-06-10T23:54:19.943851Z","shell.execute_reply.started":"2021-06-10T23:54:19.899326Z","shell.execute_reply":"2021-06-10T23:54:19.942559Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I will argue that the fundamental aspect of the competition is the *players*, since you are asked to forecast *engagement* for every player active in 2021 season. So, lets start with that.","metadata":{}},{"cell_type":"markdown","source":"# Serious Exploratory-Data-Analysis\n<hr style=\"border:0.02in solid gray\"> </hr>\n\nTo be honest, it feels sometimes like cheating, *pandas_profiling* is an automatic data profiling tools that is mounted on top of *pandas*. According to [pandas-profiling](https://github.com/pandas-profiling/pandas-profiling): *The pandas df.describe() function is great but a little basic for serious exploratory data analysis. pandas_profiling extends the pandas DataFrame with df.profile_report() for quick data analysis.*","metadata":{}},{"cell_type":"code","source":"# Lets see the data types of the columns\ndata_players.dtypes","metadata":{"execution":{"iopub.status.busy":"2021-06-10T23:54:19.947687Z","iopub.execute_input":"2021-06-10T23:54:19.94821Z","iopub.status.idle":"2021-06-10T23:54:19.959293Z","shell.execute_reply.started":"2021-06-10T23:54:19.948158Z","shell.execute_reply":"2021-06-10T23:54:19.95816Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Option 1\nplayers_profile_ds = data_players.describe()\nplayers_profile_ds","metadata":{"execution":{"iopub.status.busy":"2021-06-10T23:54:19.961002Z","iopub.execute_input":"2021-06-10T23:54:19.961449Z","iopub.status.idle":"2021-06-10T23:54:19.993488Z","shell.execute_reply.started":"2021-06-10T23:54:19.961393Z","shell.execute_reply":"2021-06-10T23:54:19.992513Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"No wonder, there are by default just 3 numeric columns, hence *describe* falls short of providing a quick summary. There are several other options of just *describe*, here is a very interesting one.","metadata":{}},{"cell_type":"code","source":"# Option 2\nplayers_profile_pp = pp.ProfileReport(data_players, title = \"players.csv Data Profile\")\nplayers_profile_pp.to_notebook_iframe()","metadata":{"execution":{"iopub.status.busy":"2021-06-10T23:54:19.994943Z","iopub.execute_input":"2021-06-10T23:54:19.995236Z","iopub.status.idle":"2021-06-10T23:54:30.866327Z","shell.execute_reply.started":"2021-06-10T23:54:19.995207Z","shell.execute_reply":"2021-06-10T23:54:30.865125Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Exploration\n<hr style=\"border:0.02in solid gray\"> </hr>\n\nAre the *birthStateProvince* and *birthCity* NAs because of those players were born outside USA ?","metadata":{}},{"cell_type":"code","source":"display(data_players.head(10))","metadata":{"execution":{"iopub.status.busy":"2021-06-10T23:54:30.870624Z","iopub.execute_input":"2021-06-10T23:54:30.871067Z","iopub.status.idle":"2021-06-10T23:54:30.897971Z","shell.execute_reply.started":"2021-06-10T23:54:30.871027Z","shell.execute_reply":"2021-06-10T23:54:30.897055Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Lets get the list of _birthCountry_, and relate to that the _birthStateProvince_ and _birthCity_, so we can see NaNs values according to each _birthCountry_","metadata":{}},{"cell_type":"code","source":"# Unique countries\ncountries = list(data_players['birthCountry'].unique())\n\n# Number of NAs in StateProvince and City\nstate_nas = [data_players[data_players['birthCountry'] == country]['birthStateProvince'].isna().sum() for country in countries]\ncity_nas = [data_players[data_players['birthCountry'] == country]['birthCity'].isna().sum() for country in countries]\ncountry_count = [len(data_players[data_players['birthCountry'] == country]) for country in countries]\n\n# Aggregated dataframe\norigins = pd.DataFrame({'birthCountry': countries, 'count_birthCountry': country_count, 'NaNs_StateProvince': state_nas, 'NaNs_City': city_nas})\n\n# Ordered dataframe\norigins = origins.sort_values(by=['birthCountry'], inplace=False, ascending=False)\ndata_players = data_players.sort_values(by=['birthCountry'], inplace=False, ascending=False)\n\n# Display results and reference\ndisplay(origins)","metadata":{"execution":{"iopub.status.busy":"2021-06-10T23:54:30.899498Z","iopub.execute_input":"2021-06-10T23:54:30.899802Z","iopub.status.idle":"2021-06-10T23:54:31.025403Z","shell.execute_reply.started":"2021-06-10T23:54:30.899772Z","shell.execute_reply":"2021-06-10T23:54:31.024365Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Well, as suspected, the NaNs do not come from players born in USA. As pandas_profiling function previously stated, there are: 545 missing values in StateProvince.","metadata":{}},{"cell_type":"code","source":"display('There are: {NaNs} missing values in StateProvince '.format(NaNs = sum(origins['NaNs_StateProvince'])), )","metadata":{"execution":{"iopub.status.busy":"2021-06-10T23:54:31.026429Z","iopub.execute_input":"2021-06-10T23:54:31.026703Z","iopub.status.idle":"2021-06-10T23:54:31.03397Z","shell.execute_reply.started":"2021-06-10T23:54:31.026676Z","shell.execute_reply":"2021-06-10T23:54:31.032796Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Interestingly enough, none where from Mexico, and since the majority of Latinos in USA are from that country, and in the North of Mexico baseball is a popular sport, it could be the case that engagement from Mexican players has interesting properties. But of course, that would be up to you to decide explore.","metadata":{}},{"cell_type":"code","source":"display(origins[origins['NaNs_StateProvince'] > 0])","metadata":{"execution":{"iopub.status.busy":"2021-06-10T23:54:31.035244Z","iopub.execute_input":"2021-06-10T23:54:31.035533Z","iopub.status.idle":"2021-06-10T23:54:31.0574Z","shell.execute_reply.started":"2021-06-10T23:54:31.035504Z","shell.execute_reply":"2021-06-10T23:54:31.055866Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Well, that is for the short version, What do you think about quick progress and iterative content in this notebook, does that make sense ?","metadata":{}}]}