{"cells":[{"metadata":{},"cell_type":"markdown","source":"# NCAA Madness Analytics\n### -- Anzhe Memphis Meng","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Content\n## * [1 Introduction](chapter1)\n## * [2 Visual EDA](chapter2)\n## * [3 Efficient Field Goal](chapter3)\n## * [4 Player Efficiency Rating](chapter4)\n## * [5 Reference](chapter5)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"\n\n# 1 Introduction\n\nWelcome to my Exploratory Data Analysis (a.k.a, EDA) for [Google Cloud & NCAA® March Madness Analytics](http://https://www.kaggle.com/c/march-madness-analytics-2020) Competition! This is a competition that requires candidates to quantify the \"madness\" of this fascinating tournament. I'm pretty sure you've at least heard [\"March Madness\"](http://https://www.ncaa.com/news/basketball-men/bracketiq/2020-04-20/what-march-madness-ncaa-tournament-explained) when you are going to read my notebook. So now let me tell you what I am going to do in the rest of the notebook.\n\nMy **goal** is illustrating the Madness with multiple python libraries, such as [pandas](http://https://pandas.pydata.org/pandas-docs/stable/index.html), [matplotib](http://https://matplotlib.org/3.2.1/index.html), [seaborn](http://https://seaborn.pydata.org/) and [plotly](http://plotly.com/) at first, and then look deeper into the relationship between a team's performance between its efficiency and their key players between 2015-2018. So I am going to utilize to two significant meters: [Efficient Field Goal Percentage](http://en.wikipedia.org/wiki/Effective_field_goal_percentage) (*abbr.* eFG%) and [Player Efficiency Rating](http://en.wikipedia.org/wiki/Player_efficiency_rating) (*abbr.* PER) to respcectively measure the teams' and players' efficiency on the court. I am going to do my research entirely focused on the Men's performance, you can feel free and try it on the Women's. I'm sure that must be pretty similar.\n\nMost of my work will be based on the [provided data](https://www.kaggle.com/c/march-madness-analytics-2020/data) from the host. But I will also apply some [self-collected data](https://www.kaggle.com/anzhemeng/ncaa-d1-finalists-ever) to improvise my illustration.\n\nFor any other details of this wonderful tournament, please refer to NCAA on [wikipedia](https://en.wikipedia.org/wiki/National_Collegiate_Athletic_Association).\n\nAlright, time's up for warming up. It's the jump-ball whistle now!\n\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# 2 Visual EDA\n\nNow we are going to see what the data basically looks like by plotting some representative graphs. This may involve mainly the teams' performance and expected performance (according to their seeds) throughout the decades.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Import libraries","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":false,"_kg_hide-output":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \nimport warnings\nwarnings.filterwarnings('ignore')\n\nimport matplotlib.pyplot as plt\nimport math\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport re\nimport seaborn as sns\nimport scipy.stats as stats\nfrom scipy.interpolate import interp1d\nimport statsmodels.api as sm\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n print(os.path.join(dirname, filename))\n# %matplotlib inline\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Import data","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":false,"_kg_hide-output":true},"cell_type":"code","source":"# team\nteams = pd.read_csv('/kaggle/input/march-madness-analytics-2020/2020DataFiles/2020DataFiles/2020-Mens-Data/MDataFiles_Stage1/MTeams.csv')\n# season\nseasons = pd.read_csv('/kaggle/input/march-madness-analytics-2020/2020DataFiles/2020DataFiles/2020-Mens-Data/MDataFiles_Stage1/MSeasons.csv')\n# seed\nseeds = pd.read_csv(r'/kaggle/input/march-madness-analytics-2020/2020DataFiles/2020DataFiles/2020-Mens-Data/MDataFiles_Stage1/MNCAATourneySeeds.csv')\n# finalists\nfinalists = pd.read_csv(r'/kaggle/input/ncaa-d1-finalists-ever/finalists.csv')\n# season compact results\nMRegularSeasonCompactResults = pd.read_csv('/kaggle/input/march-madness-analytics-2020/2020DataFiles/2020DataFiles/2020-Mens-Data/MDataFiles_Stage1/MRegularSeasonCompactResults.csv')\n# tourney compact results\nMNCAATourneyCompactResults = pd.read_csv('/kaggle/input/march-madness-analytics-2020/MDataFiles_Stage2/MNCAATourneyCompactResults.csv')\n# coaches\ncoaches = pd.read_csv('/kaggle/input/march-madness-analytics-2020/2020DataFiles/2020DataFiles/2020-Mens-Data/MDataFiles_Stage1/MTeamCoaches.csv')\n# events\nevents2015 = pd.read_csv('/kaggle/input/march-madness-analytics-2020/2020DataFiles/2020DataFiles/2020-Mens-Data/MEvents2015.csv')\nevents2016 = pd.read_csv('/kaggle/input/march-madness-analytics-2020/2020DataFiles/2020DataFiles/2020-Mens-Data/MEvents2016.csv')\nevents2017 = pd.read_csv('/kaggle/input/march-madness-analytics-2020/2020DataFiles/2020DataFiles/2020-Mens-Data/MEvents2017.csv')\nevents2018 = pd.read_csv('/kaggle/input/march-madness-analytics-2020/2020DataFiles/2020DataFiles/2020-Mens-Data/MEvents2018.csv')\nevents2019 = pd.read_csv('/kaggle/input/march-madness-analytics-2020/2020DataFiles/2020DataFiles/2020-Mens-Data/MEvents2019.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Where did the team join the league?\n\nIn order to maintain our evaluation to be unbiased, it is necessary to ensure all the inrelevant factors are the same. At least they are similar. As far as I am concerned, the season when a team participated in this tournament for the first time is quite inrelevant to our research. But we stil can have a look at it. \n\nIn the following graph, we can see that most teams joined the leaguea in 1985 which is the earliest season of our dataset and likewise, most of them are still taking part in this tournament this season. So we may research on this target with the assumption that the teams were and are facing a similar competition from other candidates every year.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"teams['FirstD1Season'].plot.hist(legend=True), teams['LastD1Season'].plot.hist(legend=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Noticed that there were some teams joining the league after 1985, so obviously there number of total teams since 1985 witnesses an rise. Then in some following situtation we can not only focus on the absolute figures but also the relative ones.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"team_amounts = []\n\nfor i in range(1985, 2020):\n team_amounts.append(len(np.unique(MRegularSeasonCompactResults.groupby('Season').get_group(i)[['WTeamID', 'LTeamID']].values)))\n\nplt.plot(seasons['Season'][:-1], team_amounts)\nplt.title(label='Number of Teams per Regular Season')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Where are the regions called in these years?\n\nBased on the charts below, in most of the time All regions' names are quite fixed except for Region X. In general, we could refer Region W to East, Region Y to Midwest and Region Z to West. Unlike them, Region X are comparably variable. But it is mostly called Midwest, West and South.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(14, 10))\n\nregions = {'RegionW': 'r', 'RegionX': 'g', 'RegionY': 'b', 'RegionZ': 'y'}\n\nfor i, key in enumerate(regions):\n plt.subplot(2, 2, i+1)\n title = key[:-1] + ' ' + key[-1]\n seasons[key].drop(axis=0, index=35).value_counts().sort_values().plot.barh(title=title, color=regions[key])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Who is Goliath?\n\nSeed Ranking is a very important index that helps the audience judge the condition of a team. Obviously, a team with higher rank in the list is more likely to win a game with an opponent whose seed ranking is lower. But who knows? The NO.1 seed was eliminated over and over again. So let's see what the data will tells us.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# look up the rank of each team in each season\nrank = seeds.merge(teams)\nrank['Rank'] = rank['Seed'].apply(lambda x : re.findall(r'([0-9]*)', x))\n\nfor i in range(len(rank['Rank'])):\n a = rank['Rank'].iloc[i]\n b = int(a[1])\n rank['Rank'].iloc[i] = b","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The ranking in each season of each team","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"rank[['Season', 'TeamName', 'Rank']]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As the following charts show, some teams seem to be better than others as their ranks are quite stable and always on the top. For example, Duke, North Carolina, Kentucky and Kansas. To be honest I'm not a big fan of NCAA but stil following [NBA](https://en.wikipedia.org/wiki/National_Basketball_Association). So speaking of these four, tons of names of superstars who graduated from them came up to me. As Isaac Newton said, \"If I have seen farther than others, it is because I was standing on the shoulders of giants.\" Michael Jordan (North Carolina), Grant Hill (Duke), Anthony Davis (Kentucky), ..., I will not deny their effort after they signed their professional contract, though it is also undoubtable that the nurture from the campuses partly make them what they are. That's an apparent sign of college Goliaths in terms of athletic basketball.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(20, 20))\n\nfor i in range(16):\n plt.subplot(4, 4, 1+i)\n seed = rank.groupby(['Rank']).get_group(1 + i)\n first4_seed = seed['TeamName'].value_counts().sort_values(ascending=False).head(4)\n if i == 0:\n first4_seed.plot.barh(title=str(i+1) + 'st seed', color='#00' + str(8000 + 1000 * i))\n elif i == 1:\n first4_seed.plot.barh(title=str(i+1) + 'nd seed', color='#00' + str(8000 + 1000 * i))\n elif i == 2:\n first4_seed.plot.barh(title=str(i+1) + 'rd seed', color='#0' + str(8000 + 1000 * i))\n else:\n first4_seed.plot.barh(title=str(i+1) + 'th seed', color='#0' + str(8000 + 1000 * i))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I prove my opinion above in the perspective of the stability. As the line plotting shows, none of the four has ranked lower than 8th, which is definitely a strong proof to their strength.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize=(12, 12))\n\ncolleges = {'North Carolina': 'c', 'Duke': 'm', 'Kentucky': 'y', 'Kansas': 'k'}\n\nfor i, key in enumerate(colleges):\n ax = fig.add_subplot(411+i)\n college = rank.groupby(['TeamName']).get_group(key)\n ax.plot(college['Season'], college['Rank'], label=key, color=colleges[key], marker='o')\n plt.gca().invert_yaxis()\n ax.legend()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Who is David?\n\nExcept these four, there are a lot of team that were dominent once. To illustrate their power, I focus on the candidacy of the national final of each year. According to the chart below, there is at least one team ranking regional NO. 1 seed. That may sound incredible, since NCAA has been famous for unpredictableness and \"Cinderella-ness\". And the most extreme situation is that the regional No.8 passed all way till the final. This has happened three times in this 35-year period (14' Kentucky, 11' Butler, 85' Vilanova).","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# only consider the finalists since 1985\nfinalists = finalists.head(35)\n\ntop_seeds = pd.merge(pd.merge(finalists, rank, left_on=['champion', 'year'], right_on=['TeamName', 'Season']),rank, \n left_on=['runner_up', 'year'], right_on=['TeamName', 'Season'], suffixes=('_champion', '_runner_up'))\n\ntop_seeds.plot.bar(x='year', y=['Rank_champion', 'Rank_runner_up'], legend='reverse')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now let's move on to the wins. You might have already guessed out, playing a home game is more likey to win.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"MRegularSeasonCompactResults['WLoc'].value_counts().plot.bar()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"ax = sns.distplot(np.log(MRegularSeasonCompactResults['WScore']), fit_kws={\"color\":\"red\"}, kde=False,\n fit=stats.gamma, hist=None, label=\"Winners' score distribution(Log Scale)\")\nax = sns.distplot(np.log(MRegularSeasonCompactResults['LScore']), fit_kws={\"color\":\"blue\"}, kde=False,\n fit=stats.gamma, hist=None, label=\"Losers' score distribution(Log Scale)\")\nax.legend()\nl1 = ax.lines[0]\nl2 = ax.lines[1]\nx1 = l1.get_xydata()[:,0]\ny1 = l1.get_xydata()[:,1]\nx2 = l2.get_xydata()[:,0]\ny2 = l2.get_xydata()[:,1]\nax.fill_between(x1,y1, color=\"red\", alpha=0.3)\nax.fill_between(x2,y2, color=\"blue\", alpha=0.3)\nprint('A winner on average scores {} more points than its opponent.'.format(math.exp(y1.mean())-math.exp(y2.mean())))\nplt.show(block=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"On average, a winner of a match usually scores 1.2 points more than the counterpart.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"MRegularSeasonCompactResults['Score_Difference'] = np.log(MRegularSeasonCompactResults['WScore'] - MRegularSeasonCompactResults['LScore'])\nax = sns.distplot(MRegularSeasonCompactResults['Score_Difference'], fit_kws={\"color\":\"green\"}, kde=False,\n fit=stats.gamma, hist=None, label=\"Score Difference Distribution (Log Scale)\")\n\nax.legend()\n\nl = ax.lines[0]\n\nx = l.get_xydata()[:,0]\ny = l.get_xydata()[:,1]\n\nax.fill_between(x,y, color=\"green\", alpha=0.3)\nprint('The average gap of a game is {}.'.format(math.exp(y.mean())))\nplt.show(block=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"When we sum up the wins of each team, it is found that out of the top 5, four teams are exactly the Duke, North Carolina, Kansas and Kentucky. Another proof to their strength.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"pd.merge(MRegularSeasonCompactResults, teams, left_on='WTeamID', right_on='TeamID')['TeamName'].value_counts().head(5).sort_values().plot.barh(title='Most winning teams(top 5)')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"However, when a game staggers and eventually need over-time to determine the winner, Big 4 are probably not the advantageous sides. Instead, if you were a plyer on the court, maybe you will pray the team on the other side is not West Vriginia, Ohio, Wyoming, VCU or Oklahoma.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"pd.merge(MRegularSeasonCompactResults, teams, left_on='WTeamID', right_on='TeamID').groupby('TeamName')['NumOT'].sum().sort_values(ascending=False).head(5).sort_values().plot.barh(title='Top 5 teams winning most OH')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"If we look at the list of finalists, we happen to notice that it seems both Goliath and David has the similar chance to get through there based on the observation that the most frequent guest to the national finals (Duke) just got in there one time more than Connecticut.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# classify the stages\nMNCAATourneyCompactResults['Stage'] = ''\nMNCAATourneyCompactResults['Stage'].loc[MNCAATourneyCompactResults['DayNum'] == 134] = 'First Four'\nMNCAATourneyCompactResults['Stage'].loc[MNCAATourneyCompactResults['DayNum'] == 135] = 'First Four'\nMNCAATourneyCompactResults['Stage'].loc[MNCAATourneyCompactResults['DayNum'] == 136] = 'First Round'\nMNCAATourneyCompactResults['Stage'].loc[MNCAATourneyCompactResults['DayNum'] == 137] = 'First Round'\nMNCAATourneyCompactResults['Stage'].loc[MNCAATourneyCompactResults['DayNum'] == 138] = 'Second Round'\nMNCAATourneyCompactResults['Stage'].loc[MNCAATourneyCompactResults['DayNum'] == 139] = 'Second Round'\nMNCAATourneyCompactResults['Stage'].loc[MNCAATourneyCompactResults['DayNum'] == 143] = 'Regional Semifinals'\nMNCAATourneyCompactResults['Stage'].loc[MNCAATourneyCompactResults['DayNum'] == 144] = 'Regional Semifinals'\nMNCAATourneyCompactResults['Stage'].loc[MNCAATourneyCompactResults['DayNum'] == 145] = 'Regional Finals'\nMNCAATourneyCompactResults['Stage'].loc[MNCAATourneyCompactResults['DayNum'] == 146] = 'Regional Finals'\nMNCAATourneyCompactResults['Stage'].loc[MNCAATourneyCompactResults['DayNum'] == 152] = 'National Semifinals'\nMNCAATourneyCompactResults['Stage'].loc[MNCAATourneyCompactResults['DayNum'] == 154] = 'National Finals'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"game_records = pd.merge(pd.merge(MNCAATourneyCompactResults, teams[['TeamID','TeamName']], left_on='WTeamID', right_on='TeamID'), teams[['TeamID','TeamName']], left_on='LTeamID', right_on='TeamID', suffixes=('_W','_L'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"game_records['TeamName_W'].loc[game_records['Stage'] == 'National Finals'].value_counts().head(5).sort_values(ascending=True).plot.barh()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Take another look at the \"Big 4\" I mentioned before. Not always could they enter the final round, even though they were highly rated. That's why NCAA fascinates us.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"colleges = {'Duke': [], 'North Carolina': [], 'Kansas': [], 'Kentucky': []}\n\nfor name, _list in colleges.items():\n for i in range(1985, 2020):\n try:\n result = game_records.groupby('Season').get_group(i).groupby('TeamName_L').get_group(name)['Stage']\n _list.append(result.iloc[0])\n except:\n if name in np.unique(game_records.groupby('Season').get_group(i)['TeamName_W']):\n _list.append('National Championship')\n else:\n _list.append('N/A')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"rounds = ['N/A', 'First Four', 'First Round', 'Second Round', 'Regional Semifinals', 'Regional Finals', \n 'National Semifinals', 'National Finals', 'National Championship']\ny_pos = np.arange(float(len(rounds)))\nfig = plt.figure(figsize=(20, 12))\n\nfor i, college in enumerate(['Duke', 'North Carolina', 'Kentucky', 'Kansas']):\n ax = fig.add_subplot(221+i)\n plt.yticks(y_pos, rounds)\n\n y = pd.Series(colleges[college]).apply(lambda x: ['N/A', 'First Four', 'First Round', 'Second Round', 'Regional Semifinals', 'Regional Finals', \n 'National Semifinals', 'National Finals', 'National Championship'].index(x))\n x = np.squeeze(pd.DataFrame(np.arange(1985.0, 2020.0)).values)\n\n lowess = sm.nonparametric.lowess(y, x, frac=.3)\n lowess_x = list(zip(*lowess))[0]\n lowess_y = list(zip(*lowess))[1]\n\n f = interp1d(lowess_x, lowess_y, bounds_error=False)\n\n xnew = [i/1. for i in range(1985, 2020)]\n ynew = f(xnew)\n\n plt.plot(x, y, '-o')\n plt.title(college)\n plt.plot(xnew, ynew, '-')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This the scatters that illustrates the relationship between a team's seed and its winning rate. Expectedly a team ranking higher wins more but there are stil outliers especially No. 15 and No.14. ","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"new_game_records = pd.merge(pd.merge(MRegularSeasonCompactResults, rank[['Season', 'TeamID', 'Rank', 'TeamName']], left_on=['Season', 'WTeamID'], right_on=['Season', 'TeamID']), rank[['Season', 'TeamID', 'Rank', 'TeamName']], left_on=['Season', 'LTeamID'], right_on=['Season', 'TeamID'], suffixes=('_W', '_L'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"totals = []\n# calculate win percentage\nfor i in range(1985, 2020):\n winDict = {k+1: 0 for k in range(16)}\n lossDict = {k+1: 0 for k in range(16)}\n wins = new_game_records.groupby('Season').get_group(i)['Rank_W'].value_counts().to_dict()\n winDict.update(wins)\n losses = new_game_records.groupby('Season').get_group(i)['Rank_L'].value_counts().to_dict()\n lossDict.update(losses)\n totalDict = {key: winDict[key]/(winDict[key]+lossDict[key]) for key in winDict.keys()}\n totals.append(totalDict)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"x = []\ny = []\nfor t in totals:\n x.extend(list(t.keys()))\n y.extend(list(t.values()))\n \nfig, ax = plt.subplots()\nax.set_xlim(17,0)\n\nlowess = sm.nonparametric.lowess(y, x, frac=.3)\nlowess_x = list(zip(*lowess))[0]\nlowess_y = list(zip(*lowess))[1]\n\nf = interp1d(lowess_x, lowess_y, bounds_error=False)\n\nxnew = [i/1. for i in range(16*35)]\nynew = f(xnew)\n\nplt.scatter(x,y,color='c')\nplt.plot(xnew, ynew, '-',color='g')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Who is the best coach?\n\nThere are only 12 coaches that once stood by a court to coach for more than 50 fixtures, and only 5 out of whom have won more than 50 matches. Among them \"Coach K\" Mike Krzyzewski must be the most successful one, ranking both NO.1 on the list of total games participated and total games won.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"coach_games = pd.merge(pd.merge(game_records[['Season','WTeamID', 'LTeamID']], coaches, left_on=['Season', 'WTeamID'], right_on=['Season', 'TeamID']), \n coaches, left_on=['Season','LTeamID'], right_on=['Season', 'TeamID'], suffixes=('_W', '_L'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"wins = dict() # collect the wins of coaches\ntotals = dict() # colect games played of coaches\n\nfor i in range(len(coach_games)):\n if coach_games['CoachName_W'].iloc[i] in wins:\n wins[coach_games['CoachName_W'].iloc[i]] += 1\n totals[coach_games['CoachName_W'].iloc[i]] += 1\n else:\n wins[coach_games['CoachName_W'].iloc[i]] = 1\n totals[coach_games['CoachName_W'].iloc[i]] = 1\n \n if coach_games['CoachName_L'].iloc[i] in totals:\n totals[coach_games['CoachName_L'].iloc[i]] += 1\n else:\n totals[coach_games['CoachName_L'].iloc[i]] = 1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"labels = 'mike_krzyzewski', 'roy_williams', 'john_calipari', 'jim_boeheim', 'rick_pitino', 'others'\nsizes = [wins['mike_krzyzewski']/sum(wins.values()), wins['roy_williams']/sum(wins.values()), wins['john_calipari']/sum(wins.values()),\n wins['jim_boeheim']/sum(wins.values()), wins['rick_pitino']/sum(wins.values()), \n 1-(wins['mike_krzyzewski']+wins['roy_williams']+wins['john_calipari']+wins['jim_boeheim']+wins['rick_pitino'])/sum(wins.values())]\nexplode = (0.5, 0.0, 0, 0,0,0) # only \"explode\" the 2nd slice (i.e. 'Hogs')\n\nfig1, ax1 = plt.subplots()\nax1.pie(sizes, explode=explode, labels=labels, autopct='%1.1f%%',\n shadow=False, startangle=90)\nax1.axis('equal') # Equal aspect ratio ensures that pie is drawn as a circle.\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"rates = dict()\n\nfor k1, v1 in wins.items():\n for k2, v2 in totals.items():\n if v2 >= 50 and k1 == k2:\n rates[k1] = v1/v2","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"rates = {k: v for k, v in sorted(rates.items(), key=lambda item: item[1], reverse=True)}\n\nprint(\"The winning percentage of the most successful coaches:\")\nfor k,v in rates.items():\n print(k, \":\", v)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3 Efficient Field Goal","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Now I am going to do some regression research. I will explore how efficient a team is defines their final apprearance in this tournament. Before we start, it is necessary to get a look at the EFG formula:\n\n\\begin{align}\nEFG\\% & = all\\ field\\ goals\\ made + \\frac{1}{2}\\times \\frac{all\\ three-pointers\\ made}{all\\ field\\ goals\\ attempted}\n\\end{align}\n\nThis equation gives more weights to the three pointers, so a team that is good at three-point shooting will have an advantage.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"According to my illustration, there is no apparent connection between this index and the performance shown. The most extreme example is in 2018-2019 season when the runner-up's efficiency is almost at the bottom of the list while the champion barely among the mainstream. My understanding to this is that this index only takes the offense performance into consideration while defense also plays a great role in a game. Meanwhile, it is possible that although one team plays well except the game they are eliminated. That leads to the phenomon where some seemingly excellent teams are knocked out early in the competition.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Anyway, it explains to us again the charm of March Madness, where an underdog is not always a loser.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"team_stat_2015 = dict()\nteam_stat_2016 = dict()\nteam_stat_2017 = dict()\nteam_stat_2018 = dict()\nteam_stat_2019 = dict()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def archive_team_stat(events, team_stat):\n for team in events['EventTeamID'].unique():\n if team not in team_stat:\n team_stat[team] = dict()\n\n team_stat[team]['matches'] = list()\n team_stat[team]['field_goals'] = 0\n team_stat[team]['field_goals_attempted'] = 0\n team_stat[team]['three_pointers'] = 0\n team_stat[team]['three_pointers_attempted'] = 0\n\n info = events.groupby('EventTeamID').get_group(team)\n\n for i in range(info.shape[0]):\n if info['DayNum'].iloc[i] not in team_stat[team]['matches']:\n team_stat[team]['matches'].append(info['DayNum'].iloc[i])\n elif info['EventType'].iloc[i] == 'made2':\n team_stat[team]['field_goals'] += 1\n team_stat[team]['field_goals_attempted'] += 1\n elif info['EventType'].iloc[i] == 'miss2':\n team_stat[team]['field_goals_attempted'] += 1\n elif info['EventType'].iloc[i] == 'made3':\n team_stat[team]['three_pointers'] += 1\n team_stat[team]['three_pointers_attempted'] += 1\n elif info['EventType'].iloc[i] == 'miss3':\n team_stat[team]['three_pointers_attempted'] += 1\n \n for team, stat in team_stat.items():\n try:\n team_stat[team]['EFG'] = team_stat[team]['field_goals'] + team_stat[team]['three_pointers'] + 0.5 * team_stat[team]['three_pointers'] / (team_stat[team]['field_goals_attempted'] + team_stat[team]['three_pointers_attempted'])\n team_stat[team]['EFG'] = team_stat[team]['EFG'] / len(team_stat[team]['matches'])\n except:\n team_stat[team]['EFG'] = 0\n return team_stat","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"team_stats = [team_stat_2015, team_stat_2016, team_stat_2017, team_stat_2018, team_stat_2019]\nevents = [events2015, events2016, events2017, events2018, events2019]\n\nfor i in range(len(team_stats)):\n team_stats[i] = archive_team_stat(events[i], team_stats[i])\n # sort by the value of EFG\n team_stats[i] = {k: v for k, v in sorted(team_stats[i].items(), key=lambda item: item[1]['EFG'], reverse=True)}\n team_stats[i] = {k: v for k, v in team_stats[i].items() if k not in np.unique(game_records.groupby('Season').get_group(i+2015)[['TeamID_W', 'TeamID_L']])}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"efficiency_vs_performance_2015 = pd.DataFrame(columns=['TeamID', 'TeamName', 'EFG', 'Stage'])\nefficiency_vs_performance_2016 = pd.DataFrame(columns=['TeamID', 'TeamName', 'EFG', 'Stage'])\nefficiency_vs_performance_2017 = pd.DataFrame(columns=['TeamID', 'TeamName', 'EFG', 'Stage'])\nefficiency_vs_performance_2018 = pd.DataFrame(columns=['TeamID', 'TeamName', 'EFG', 'Stage'])\nefficiency_vs_performance_2019 = pd.DataFrame(columns=['TeamID', 'TeamName', 'EFG', 'Stage'])\n\nevsp = [efficiency_vs_performance_2015, efficiency_vs_performance_2016, efficiency_vs_performance_2017, efficiency_vs_performance_2018, efficiency_vs_performance_2019]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def efficiency_archive(index, df, team_stat):\n for team in np.unique(game_records.groupby('Season').get_group(index+2015)[['TeamID_W', 'TeamID_L']]):\n try:\n team_name = game_records.groupby(['Season', 'TeamID_W']).get_group((index+2015, team))['TeamName_W'].iloc[0]\n except:\n team_name = game_records.groupby(['Season', 'TeamID_L']).get_group((index+2015, team))['TeamName_L'].iloc[0]\n# print(team_name)\n try:\n win_day = game_records.groupby(['Season', 'TeamID_W']).get_group((index+2015, team))['DayNum'].max()\n except:\n win_day = 0\n try:\n loss_day = game_records.groupby(['Season', 'TeamID_L']).get_group((index+2015, team))['DayNum'].max()\n except:\n loss_day = 0\n# print(max(loss_day, win_day))\n if max(loss_day, win_day) == 134:\n stage = 'First Four'\n elif max(loss_day, win_day) == 135:\n stage = 'First Four'\n elif max(loss_day, win_day) == 136:\n stage = 'First Round' \n elif max(loss_day, win_day) == 137:\n stage = 'First Round'\n elif max(loss_day, win_day) == 138:\n stage = 'Second Round'\n elif max(loss_day, win_day) == 139:\n stage = 'Second Round'\n elif max(loss_day, win_day) == 143:\n stage = 'Regional Semifinals'\n elif max(loss_day, win_day) == 144:\n stage = 'Regional Semifinals'\n elif max(loss_day, win_day) == 145:\n stage = 'Regional Finals'\n elif max(loss_day, win_day) == 146:\n stage = 'Regional Finals'\n elif max(loss_day, win_day) == 152:\n stage = 'National Semifinals'\n elif max(loss_day, win_day) == 154:\n if loss_day == 154:\n stage = 'National Finals'\n else:\n stage = 'National Championship'\n try:\n new_row = pd.DataFrame({\n 'TeamID': [team],\n 'TeamName': [team_name],\n 'EFG': [team_stat[team]['EFG']],\n 'Stage': [stage]\n })\n except:\n if index == 0:\n new_row = pd.DataFrame({\n 'TeamID': [team],\n 'TeamName': [team_name],\n 'EFG': [team_stat_2015[team]['EFG']],\n 'Stage': [stage]\n })\n elif index == 1:\n new_row = pd.DataFrame({\n 'TeamID': [team],\n 'TeamName': [team_name],\n 'EFG': [team_stat_2016[team]['EFG']],\n 'Stage': [stage]\n })\n elif index == 2:\n new_row = pd.DataFrame({\n 'TeamID': [team],\n 'TeamName': [team_name],\n 'EFG': [team_stat_2017[team]['EFG']],\n 'Stage': [stage]\n })\n elif index == 3:\n new_row = pd.DataFrame({\n 'TeamID': [team],\n 'TeamName': [team_name],\n 'EFG': [team_stat_2018[team]['EFG']],\n 'Stage': [stage]\n })\n elif index == 4:\n new_row = pd.DataFrame({\n 'TeamID': [team],\n 'TeamName': [team_name],\n 'EFG': [team_stat_2019[team]['EFG']],\n 'Stage': [stage]\n })\n # print(new_row)\n df = pd.concat([df, new_row], ignore_index=True)\n \n return df","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"for i in range(len(evsp)):\n evsp[i] = efficiency_archive(i, evsp[i], team_stats[i])","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"rounds = ['First Four', 'First Round', 'Second Round', 'Regional Semifinals', 'Regional Finals', \n 'National Semifinals', 'National Finals', 'National Championship']\ny_pos = np.arange(float(len(rounds)))\nfig = plt.figure(figsize=(15, 80))\n\nfor i in range(len(evsp)):\n ax = fig.add_subplot(511+i)\n y = evsp[i]['Stage'].apply(lambda x: ['First Four', 'First Round', 'Second Round', 'Regional Semifinals', 'Regional Finals', \n 'National Semifinals', 'National Finals', 'National Championship'].index(x))\n x = evsp[i]['EFG']\n\n chart = sns.regplot(x, y, lowess=True)\n labels = [item.get_text() for item in chart.get_yticklabels()]\n for j in range(len(rounds)):\n labels[1+j] = rounds[j]\n chart.set_yticklabels(labels)\n plt.title(str(2015+i))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Altough generally there is no big difference between the teams in terms of EFG, EFG of every team varied from game to game. Let's see how the teams performed at the fixture that they were eliminated.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def calculate_final_EFG(df, event):\n df.insert(2, \"Final_EFG\", [0 for _ in range(df.shape[0])], True) \n for team in event['EventTeamID'].unique():\n if team != 0:\n field_goals = 0\n field_goals_attempted = 0\n three_pointers = 0\n three_pointers_attempted = 0\n\n try:\n last_loss = max(event.groupby('LTeamID').get_group(team)['DayNum'])\n except:\n # a team did not lose\n last_loss = 0\n\n try:\n last_win = max(event.groupby('WTeamID').get_group(team)['DayNum'])\n except:\n # a team did not win\n last_win = 0\n\n final_match = max(last_loss, last_win)\n try:\n info = event.groupby(['EventTeamID', 'DayNum']).get_group((team, final_match))\n for i in range(info.shape[0]):\n if info['EventType'].iloc[i] == 'made2':\n field_goals += 1\n field_goals_attempted += 1\n elif info['EventType'].iloc[i] == 'miss2':\n field_goals_attempted += 1\n elif info['EventType'].iloc[i] == 'made3':\n three_pointers += 1\n three_pointers_attempted += 1\n elif info['EventType'].iloc[i] == 'miss3':\n three_pointers_attempted += 1\n\n new_EFG = 0\n new_EFG = field_goals + three_pointers + 0.5 * three_pointers / (field_goals_attempted + three_pointers_attempted)\n df['Final_EFG'].loc[df['TeamID'] == team] = new_EFG\n except:\n print('ERROR!')\n \n return df","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"for i in range(len(evsp)):\n evsp[i] = calculate_final_EFG(evsp[i], events[i])","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"losers = pd.DataFrame(columns=['TeamID', 'TeamName', 'Final_EFG', 'EFG', 'Stage'])\n\nfor i in range(len(evsp)):\n # eliminate dirty data\n l = evsp[i].drop(evsp[i][evsp[i]['Final_EFG'] == 0].index)\n # eliminate champions' data\n if i == 0:\n l = l.drop(l[l['TeamName'] == 'Duke'].index)\n elif i == 1:\n l = l.drop(l[l['TeamName'] == 'Villanova'].index)\n elif i == 2:\n l = l.drop(l[l['TeamName'] == 'North Carolina'].index)\n elif i == 3:\n l = l.drop(l[l['TeamName'] == 'Villanova'].index)\n elif i == 4:\n l = l.drop(l[l['TeamName'] == 'Virginia'].index)\n \n losers = pd.concat([losers, l], ignore_index=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the following image, we can see that on average the eliminated team were likely to perform worse in terms of EFG (approximately 1.04 less than average). Considering EFG reflects the offence of a team, so we can safely conclude that the side good at defence is the advantageous one.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"losers['EFG_Difference'] = losers['EFG'] - losers['Final_EFG']\nax = sns.distplot(losers['EFG_Difference'], fit_kws={\"color\":\"green\"}, kde=False,\n fit=stats.gamma, hist=None, label=\"EFG Difference\")\n\nax.legend()\n\nl = ax.lines[0]\n\nx = l.get_xydata()[:,0]\ny = l.get_xydata()[:,1]\n\nax.fill_between(x,y, color=\"green\", alpha=0.3)\nprint('The average difference of EFG from average performance is {}.'.format(math.exp(y.mean())))\nplt.show(block=False)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"champions = pd.DataFrame(columns=['TeamID', 'TeamName', 'Final_EFG', 'EFG', 'Stage'])\n\nchampions = pd.concat([champions, evsp[0][evsp[0]['TeamName'] == 'Duke']])\nchampions = pd.concat([champions, evsp[1][evsp[1]['TeamName'] == 'Villanova']])\nchampions = pd.concat([champions, evsp[2][evsp[2]['TeamName'] == 'North Carolina']])\nchampions = pd.concat([champions, evsp[3][evsp[3]['TeamName'] == 'Villanova']])\nchampions = pd.concat([champions, evsp[4][evsp[4]['TeamName'] == 'Virginia']])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":" If we just have a look at the champions in these years, EFG of them are not always different from before. Perhaps they just paid more attention to defence in the final and it worked. Only Villanova is an exception. It is the only college that won the title twice in the 5 years and both time their EFG was much higher their own average.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"ind = np.arange(5)\nwidth = 0.35 \nplt.bar(ind, champions['Final_EFG'], width, label='EFG in Final')\nplt.bar(ind + width, champions['EFG'], width,\n label='Average EFG')\n\nplt.ylabel('EFG')\nplt.title('EFG of National Champions (2015-2019)')\n\nplt.xticks(ind + width / 2, ('2015\\nDuke', '2016\\nVillanova', '2017\\nNorth Carolina', '2018\\nVillanova', '2019\\nVirginia'))\nplt.legend(loc='best')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 4 Player Efficiency Rating","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Although basketball is a teamwork sport, one or two outstanding player is able to determine the games from time to time. So hopefully we can discover the connection between a team's overall performance and the most pivotal player's individual performance. In NBA, there has been a matured strategy to measure a player's contribution on the court, including +/- value and PER. Since what I desire is the player's efficiency, PER is more preferable choice. However, since the PER and created by [John Hollinger](https://en.wikipedia.org/wiki/John_Hollinger) and utilized by NBA is a very complex stuff. In the following part of this chapter, I am going to implement a rough PER formula.\n\nI call this PER \"pure PER\" because it merely cares about the most basic statistics and does not involve any complicated calculation. It is show as below.\n\n\\begin{align}\npure\\ PER\\% & = points\\ per\\ game + rebounds\\ per\\ game + assists\\ per\\ game + steals\\ per\\ game + blocks\\ per\\ game - turnovers\\ per\\ game - missed\\ field\\ goals\\ per\\ game + missed\\ free\\ throws\\ per\\ game \n\\end{align}","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# 5 Reference\n\n\\[1] [Mathletics: How Gamblers, Managers, and Sports Enthusiasts Use Mathematics in Baseball, Basketball, and Football, Wayne L. Winston](https://www-jstor-org.ezproxy.bu.edu/stable/j.ctt7sj9q)","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}