{"cells":[{"metadata":{},"cell_type":"markdown","source":"# 2020 March Madness : Exploratory notebook\n\nFor this kernel, I forked the excellent [Exploratory kernel](https://www.kaggle.com/robikscube/2020-march-madness-data-lets-take-a-look) of Rob Mulla for data extraction. I have replaced its stats and graphs by mine, which highlight other aspects of teams and players. Feel free to upvote or comment this kernel if you like it!\n","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pylab as plt\nfrom matplotlib.patches import Circle, Rectangle, Arc\nimport seaborn as sns\nplt.style.use('fivethirtyeight')\nplt.rcParams['font.family'] = 'serif'\nplt.rcParams['font.serif'] = 'Ubuntu'\nplt.rcParams['font.monospace'] = 'Ubuntu Mono'\nplt.rcParams['font.size'] = 10\nplt.rcParams['axes.labelsize'] = 10\nplt.rcParams['axes.labelweight'] = 'bold'\nplt.rcParams['xtick.labelsize'] = 8\nplt.rcParams['ytick.labelsize'] = 8\nplt.rcParams['legend.fontsize'] = 10\nplt.rcParams['figure.titlesize'] = 12\nmypal = plt.rcParams['axes.prop_cycle'].by_key()['color'] # Grab the color pal\nimport os\nimport gc\n\nMENS_DIR = '../input/google-cloud-ncaa-march-madness-2020-division-1-mens-tournament'\nWOMENS_DIR = '../input/google-cloud-ncaa-march-madness-2020-division-1-womens-tournament'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Team Data\n**MTeams & WTeams**\n\nTeam name and Team ID, first and last D1 Season. Sorting by the `FirstD1Season` column we can see some of the newest teams in D1 basketball. Welcome to D1 Merrimack! Cool mascot.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"MTeams = pd.read_csv(f'{MENS_DIR}/MDataFiles_Stage1/MTeams.csv')\nMTeams.sort_values('FirstD1Season', ascending=False).head(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Womens' data does not contain years joined :(\nWTeams = pd.read_csv(f'{WOMENS_DIR}/WDataFiles_Stage1/WTeams.csv')\nWTeams.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## MSeasons.csv & WSeasons.csv\nThese files identify the different seasons included in the historical data, along with certain season-level properties.\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"MSeasons = pd.read_csv(f'{MENS_DIR}/MDataFiles_Stage1/MSeasons.csv')\nWSeasons = pd.read_csv(f'{WOMENS_DIR}/WDataFiles_Stage1/WSeasons.csv')\nMSeasons.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Regular Season Results\n**MRegularSeasonCompactResults.csv & WRegularSeasonCompactResults.csv, MNCAATourneyCompactResults.csv & WNCAATourneyCompactResults**\n\nThese files identify the game-by-game NCAA® tournament results for all seasons of historical data.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"MRegularSeasonCompactResults = pd.read_csv(f'{MENS_DIR}/MDataFiles_Stage1/MRegularSeasonCompactResults.csv')\nWRegularSeasonCompactResults = pd.read_csv(f'{WOMENS_DIR}/WDataFiles_Stage1/WRegularSeasonCompactResults.csv')\nMNCAATourneyCompactResults = pd.read_csv(f'{MENS_DIR}/MDataFiles_Stage1/MNCAATourneyCompactResults.csv')\nWNCAATourneyCompactResults = pd.read_csv(f'{WOMENS_DIR}/WDataFiles_Stage1/WNCAATourneyCompactResults.csv')\nMAllResults = MRegularSeasonCompactResults.append(MNCAATourneyCompactResults)\nWAllResults = WRegularSeasonCompactResults.append(WNCAATourneyCompactResults)\nMAllResults.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Lets Add the winning and losing team names to the results\nMAllResults = \\\n    MAllResults \\\n    .merge(MTeams[['TeamName', 'TeamID']],\n           left_on='WTeamID',\n           right_on='TeamID',\n           validate='many_to_one') \\\n    .drop('TeamID', axis=1) \\\n    .rename(columns={'TeamName': 'WTeamName'}) \\\n    .merge(MTeams[['TeamName', 'TeamID']],\n           left_on='LTeamID',\n           right_on='TeamID') \\\n    .drop('TeamID', axis=1) \\\n    .rename(columns={'TeamName': 'LTeamName'})\n\nWAllResults = \\\n    WAllResults \\\n    .merge(WTeams[['TeamName', 'TeamID']],\n           left_on='WTeamID',\n           right_on='TeamID',\n           validate='many_to_one') \\\n    .drop('TeamID', axis=1) \\\n    .rename(columns={'TeamName': 'WTeamName'}) \\\n    .merge(WTeams[['TeamName', 'TeamID']],\n           left_on='LTeamID',\n           right_on='TeamID') \\\n    .drop('TeamID', axis=1) \\\n    .rename(columns={'TeamName': 'LTeamName'})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"WAllResults.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's see how the point scored per game evolved over time:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"WAllResults['TScore'] = WAllResults['WScore'] + WAllResults['LScore']\nWScores = WAllResults.groupby('Season')['TScore'].sum() / WAllResults.groupby('Season')['TScore'].count()\nMAllResults['TScore'] = MAllResults['WScore'] + MAllResults['LScore']\nMScores = MAllResults.groupby('Season')['TScore'].sum() / MAllResults.groupby('Season')['TScore'].count()\nplt.plot(WScores, 'red', label = 'Women')\nplt.plot(MScores, 'blue', label = 'Men')\nplt.title('Points scored per game')\nleg=plt.legend(loc='best')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here, we see that Men and Women team grossly have the same evolution, although Mens score more points per game.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Analysis of teams' statistics","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_stats(df):\n    df['DiffScore'] = df['WScore'] - df['LScore']\n    df['counter'] = 1\n    games_won = df.groupby('WTeamName')['counter'].count()\n    games_lost = df.groupby('LTeamName')['counter'].count()\n    games_won_2015 = df[df['Season'] >= 2015].groupby('WTeamName')['counter'].count()\n    games_lost_2015 = df[df['Season'] >= 2015].groupby('LTeamName')['counter'].count()\n    Points_scored_won = df.groupby('WTeamName')['WScore'].sum()\n    Points_scored_loss = df.groupby('LTeamName')['LScore'].sum()\n    Points_scored_against_won = df.groupby('WTeamName')['LScore'].sum()\n    Points_scored_against_loss = df.groupby('LTeamName')['WScore'].sum()\n    Diff_W = df.groupby('WTeamName')['DiffScore'].mean()\n    Diff_L = - df.groupby('LTeamName')['DiffScore'].mean()\n    games_played = pd.concat([games_won, games_lost, games_won_2015, games_lost_2015, Diff_W, Diff_L, Points_scored_won, Points_scored_loss, Points_scored_against_won, Points_scored_against_loss], axis = 1, sort = True)\n    games_played.columns = ['Games_W', 'Games_L', 'Games_W_2015', 'Games_L_2015', 'Diff_W', 'Diff_L', 'Points_scored_W','Points_scored_L','Points_scored_against_W','Points_scored_against_L',]\n    \n    games_played['Games_Total'] = games_played['Games_L'] + games_played['Games_W']\n    games_played['Games_Total_2015'] = games_played['Games_L_2015'] + games_played['Games_W_2015']\n    games_played['Winning_Rate'] = games_played['Games_W'] * 100 / games_played['Games_Total']\n    games_played['Avg_Diff'] = (games_played['Diff_W'] * games_played['Games_W'] + games_played['Diff_L'] * games_played['Games_L']) / games_played['Games_Total']\n    games_played['Avg_Points_scored'] = (games_played['Points_scored_W'] + games_played['Points_scored_L']) / games_played['Games_Total']\n    games_played['Avg_Points_scored_against'] = (games_played['Points_scored_against_W'] + games_played['Points_scored_against_L']) / games_played['Games_Total']\n    games_played.drop(['Diff_W', 'Diff_L', 'Points_scored_W', 'Points_scored_L', 'Points_scored_against_W', 'Points_scored_against_L'], axis = 1, inplace  = True)\n    \n    return games_played\n\nMgames_played = build_stats(MAllResults)\nWgames_played = build_stats(WAllResults)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"Mgames_played.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize = (12,4)) \nplt.subplot(121)\n# REPRENDRE ICI\nMgames_played.sort_values('Winning_Rate')['Winning_Rate'].tail(10).plot(kind='barh', title = 'Men Teams with the highest % of win', cmap = plt.get_cmap('tab20c'))\nplt.subplot(122)\nWgames_played.sort_values('Winning_Rate')['Winning_Rate'].tail(10).plot(kind='barh', title = 'Women Teams with the highest % of win', style = 'ggplot')\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize = (12,4))\nplt.subplot(121)\nMgames_played.sort_values('Games_Total')['Games_Total'].tail(10).plot(kind='barh', title = 'Men Teams with most games played')\nplt.subplot(122)\nWgames_played.sort_values('Games_Total')['Games_Total'].tail(10).plot(kind='barh', title = 'Women Teams with most games played')\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize = (12,8))\nplt.subplot(221)\nMgames_played.sort_values('Avg_Diff')['Avg_Diff'].tail(10).plot(kind='barh', title = 'Men Teams with the highest Avg Score Difference')\nplt.subplot(222)\nWgames_played.sort_values('Avg_Diff')['Avg_Diff'].tail(10).plot(kind='barh', title = 'Women Teams with the highest Avg Score Difference')\nplt.subplot(223)\nMgames_played.sort_values('Avg_Diff')['Avg_Diff'].head(10).plot(kind='barh', title = 'Men Teams with the lowest Avg Score Difference')\nplt.subplot(224)\nWgames_played.sort_values('Avg_Diff')['Avg_Diff'].head(10).plot(kind='barh', title = 'Women Teams with the lowest Avg Score Difference')\n\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(12,8))\nplt.subplot(221)\nMgames_played.sort_values('Avg_Points_scored')['Avg_Points_scored'].tail(10).plot(kind='barh', title = 'Most offensive Men teams')\nplt.subplot(222)\nWgames_played.sort_values('Avg_Points_scored')['Avg_Points_scored'].tail(10).plot(kind='barh', title = 'Most offensive Women teams')\nplt.subplot(223)\nMgames_played.sort_values('Avg_Points_scored_against')['Avg_Points_scored_against'].tail(10).plot(kind='barh', title = 'Most defensive Men teams')\nplt.subplot(224)\nWgames_played.sort_values('Avg_Points_scored_against')['Avg_Points_scored_against'].head(10).plot(kind='barh', title = 'Most defensive Women teams')\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Event Data\n\nEach MEvents & WEvents file lists the play-by-play event logs for more than 99.5% of games from that season.\nEach event is assigned to either a team or a single one of the team's players.\nThus if a basket is made by one player and an assist is credited to a second player,\nthat would show up as two separate records. The players are listed by PlayerID within the xPlayers.csv file.\n\nMens Event Files:\n- MEvents2015.csv, MEvents2016.csv, MEvent2017.csv, MEvents2018.csv, MEvents2019.csv\n\nWomens Event Files:\n- WEvents2015.csv, WEvents2016.csv, WEvents2017.csv, WEvents2018.csv, WEvents2019.csv\n\nWe can read in all files and combine into one huge dataframe, one for womens and one for mens.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"mens_events = []\nfor year in [2015, 2016, 2017, 2018, 2019]:\n    mens_events.append(pd.read_csv(f'{MENS_DIR}/MEvents{year}.csv'))\nMEvents = pd.concat(mens_events)\nprint(MEvents.shape)\nMEvents.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"womens_events = []\nfor year in [2015, 2016, 2017, 2018, 2019]:\n    womens_events.append(pd.read_csv(f'{WOMENS_DIR}/WEvents{year}.csv'))\nWEvents = pd.concat(womens_events)\nprint(WEvents.shape)\nWEvents.head()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"del mens_events\ndel womens_events\ngc.collect()\n\nMEvents = MEvents.merge(MTeams[['TeamName', 'TeamID']],\n               left_on='EventTeamID',\n               right_on='TeamID',\n               validate='many_to_one')\nWEvents = WEvents.merge(WTeams[['TeamName', 'TeamID']],\n               left_on='EventTeamID',\n               right_on='TeamID',\n               validate='many_to_one')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Compare team's radar charts\nHere I wanted to find a common ground to compare all teams regarding different characteristics. I plan to deepen this analysis, but here is a radar chart graph that compares each team's characteristics from 0 (minimum value) to 1 (maximum).","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def add_events(df, games_played):\n    df['counter'] = 1\n    for ev in df['EventType'].unique():\n        games_played['{}_Count'.format(ev)] = df[df['EventType'] == ev].groupby('TeamName')['counter'].count()\n        games_played['{}_Avg'.format(ev)] = games_played['{}_Count'.format(ev)] / games_played['Games_Total_2015']\n    games_played.head()\n    games_played['Fair Play'] = (games_played['foul_Avg'] - games_played['foul_Avg'].min()) / (games_played['foul_Avg'].max() - games_played['foul_Avg'].min())\n    games_played['acc'] = (games_played['made1_Count'] + games_played['made2_Count'] + games_played['made3_Count']) / (games_played['made1_Count'] + games_played['made2_Count'] + games_played['made3_Count'] + games_played['miss1_Count'] + games_played['miss2_Count'] + games_played['miss3_Count'])\n    games_played['Accuracy'] = (games_played['acc'] - games_played['acc'].min()) / (games_played['acc'].max() - games_played['acc'].min())\n    games_played['Far-shooter'] = (games_played['made3_Avg'] - games_played['made3_Avg'].min()) / (games_played['made3_Avg'].max() - games_played['made3_Avg'].min())\n    games_played['Block'] = (games_played['block_Avg'] - games_played['block_Avg'].min()) / (games_played['block_Avg'].max() - games_played['block_Avg'].min())\n    games_played['Steal'] = (games_played['steal_Avg'] - games_played['steal_Avg'].min()) / (games_played['steal_Avg'].max() - games_played['steal_Avg'].min())\n    \n    return games_played\n\nMgames_played = add_events(MEvents, Mgames_played)\nWgames_played = add_events(WEvents, Wgames_played)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Code adapted from https://typewind.github.io/2017/09/29/radar-chart/\nlabels = ['Fair Play', 'Accuracy', 'Far-shooter', 'Block', 'Steal']\nplt.figure(figsize = (12,4))\n\ndef build_radar(team, num_subplot):\n    stats = Mgames_played.loc[team][labels].values\n    angles=np.linspace(0, 2*np.pi, len(labels), endpoint=False) # Set the angle\n    stats=np.concatenate((stats,[stats[0]]))\n    angles=np.concatenate((angles,[angles[0]]))\n    fig=plt.figure()\n    ax = fig.add_subplot(num_subplot, polar=True)   # Set polar axis\n    ax.plot(angles, stats, 'o-', linewidth=2)  # Draw the plot (or the frame on the radar chart)\n    ax.fill(angles, stats, alpha=0.25)  #Fulfill the area\n    ax.set_thetagrids(angles * 180/np.pi, labels)  # Set the label for each axis\n    ax.set_title(Mgames_played.loc[team].name) \n    ax.set_rlim(0,1)\n    ax.grid(True)\n\n\nbuild_radar('Duke', 121)\nbuild_radar('Longwood', 122)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Team ranking\nTo visualize the team ranks, I had to make some choices. First, I have only worked on the **Massey Ranking**, as it seemed quite reliable. Then, I have only plotted the ranking curves of the teams that appeared **at least once in the top 3 ranking in 2019**. Finally, I made an approximation of the rank of each team in each **third of a season**. This gave me the graph that you can see at the end of this section! \n\nThe ranking file is only available for men teams.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"MMassey = pd.read_csv(f'{MENS_DIR}/MDataFiles_Stage1/MMasseyOrdinals.csv')\nMMassey = MMassey.merge(MTeams[['TeamName', 'TeamID']],\n               left_on='TeamID',\n               right_on='TeamID',\n               validate='many_to_one')\n\n\nbest_teams = MMassey[(MMassey['SystemName'] == 'MAS') \n                      & (MMassey['OrdinalRank'] <= 3) \n                      & (MMassey['Season'] == 2019)]['TeamName'].unique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"rank_df = pd.DataFrame(columns = ['TeamName', 'OrdinalRank', 'Season', 'Temp'])\nfor season in MMassey['Season'].unique():\n    k = 0\n    counter = 0\n    for i in np.linspace(0, 133, 3, dtype = int):\n        rank_series = pd.DataFrame(MMassey[(MMassey['SystemName'] == 'MAS')\n                      & (MMassey['Season'] == season)\n                      & (MMassey['RankingDayNum'] <= i) \n                      & (MMassey['RankingDayNum'] >= k)\n                      & (MMassey['TeamName'].isin(best_teams))].groupby('TeamName', as_index=False)['OrdinalRank'].mean())\n        rank_series['Season'] = season\n        rank_series['Temp'] = counter\n        k = i\n        counter += 1\n        rank_df = rank_df.append(rank_series)\nrank_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ranking_table = pd.pivot_table(rank_df,\n               values = 'OrdinalRank',\n               index = 'TeamName',\n               columns = ['Season', 'Temp'],\n               aggfunc= (lambda x: x))\nranking_table.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ranking_table.T.plot(legend = True, \n                     colormap = 'Set1',\n                     figsize = (10,10), \n                     title = 'Average ranking of last years\\' best teams',\n                     linewidth = 1)\nplt.gca().invert_yaxis()\nplt.ylim(bottom=100, top = 0)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This shows how quick the ranking changes ! Let's note that Duke and Kansas have been in the top ranks for more than 15 years.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}