{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Uncovering March Madness® Cinderella Stories\n## Google Cloud & NCAA® March Madness Analytics 2020\n\n\nKaggle Analytics Prediction Competition\n
April 30, 2020\n\n### CONTENTS OF THIS FILE\n\n- **[I. Definition](#I.-Definition)**\n - [1.1. Project Overview](#1.1.-Project-Overview)\n - [1.2. Problem Statement](#1.2.-Problem-Statement)\n - [1.3. Methodology](#1.3.-Methodology)\n- **[II. Implementation](#II.-Implementation)**\n - [2.1. The Basics, 1985-2020](#2.1.-The-Basics,-1985-2020)\n - [2.2. Geography, 2010-2020](#2.2.-Geography,-2010-2020)\n - [2.3. Team Box Scores, 2003-2019](#2.3.-Team-Box-Scores,-2003-2019)\n - [2.4. Play-by-play, 2015-2020](#2.4.-Play-by-play,-2015-2020)\n - [2.5. Individual Statistics, 2015-2020](#2.5.-Individual-Statistics,-2015-2020)\n - [2.6. Public Rankings, 2003-2020](#2.6.-Public-Rankings,-2003-2020)\n - [2.7. Prediction Experiment](#2.7.-Prediction-Experiment)\n- **[III. Results](#III.-Results)**\n - [3.1. General Findings](#3.1.-General-Findings)\n - [3.2. Uncovering Cinderellaness](#3.2.-Uncovering-Cinderellaness)\n - [3.3. Predicting Cinderellas](#3.3.-Predicting-Cinderellas)\n- [References](#References)\n\n\n---\n#### A Note to the Reader\n\nEditorial \"we\" is used in place of \"I\" and in the meaning of \"the author and the reader\". I recommend reading the [I. Definition](#I.-Definition) and [III. Results](#III.-Results) sections before the [II. Implementation](#II.-Implementation).\n\n---\n### ABSTRACT\n\nOur analysis aimed to define common features and specific trends among \"Cinderella\" teams in NCAA® men's basketball. In this context, Cinderella was defined as any basketball team seeded 10th or worse that has advanced to the Round 3 in NCAA® tournament. We have divided all the remaining teams into two more categories - Top and Ordinary.\n\nWe explored, filtered and analyzed NCAA® data across different dimensions and used descriptive statistics and exploratory visualizations to summarize main characteristics about the data in general, and in particular about \"Cinderellaness\" as our target of interest.\n\nOur analysis demonstrated that a typical Cinderella team is ranked between 20 and 65 in a pre-tournament rankings of a popular ranking systems. Cinderellas are good at shooting 2-pointers in the regular season, but not so much in the tournaments. The opposite is true for the three-point goals - Cinderellas have the highest three-point field goal ratio in NCAA® tournaments of all team categories. Cinderellas are successful in defensive rebounding and will likely have a positive Rebound Margin in the regular season games. They typically win with a high scoring margin in Round 2 of NCAA® tournament, but it is harder for them to keep it as high in the later rounds.\n\nFor this research, we have trained eXtreme Gradient Boosting (XGBoost) machine learning algorithm to predict which team had the best potential to become a Cinderella before the March Madness was canceled. Our model predicted that ETSU (East Tennessee State University) was the most likely candidate for a Cinderella team of the 2020 season.\n\n---\n\n# I. Definition\n\n## 1.1. Project Overview\n\n**Project Origin**\n\nEach season there are thousands of men's and women's NCAA® basketball games played between Division I teams, culminating in March Madness®, the national championship tournaments that start in the middle of March [[1]](#References). The men's and women's NCAA basketball tournaments are beloved American sports traditions. These are single-elimination tournaments, which means that the championship team has to win at least six games in a row to claim the title. This high-stakes environment — plus the chance to witness a crazy \"Cinderella-story\" upset, gives the tournament its March Madness® nickname [[4]](#References).\n\nThe challenge of the \"Google Cloud & NCAA® March Madness Analytics\" competition, sponsored by [Google Cloud](https://cloud.google.com/) and hosted by [Kaggle](https://www.kaggle.com/), is to present an **exploratory analysis** of the March Madness® using a Kaggle Notebook.\n\n**Prerequisite Knowledge**\n\nIn this study, we assume that the reader is familiar with the basic [NCAA® men's basketball rules](http://www.ncaa.org/playing-rules/mens-basketball-rules-game) and terminology. For those new to basketball, we recommend [[4]](#References) and [[26]](#References) for a quick introduction.\n\n**Input Data**\n\nThe input NCAA® data is provided for this competition and is available from the competition website. The data is about college basketball games and teams and is divided into 6 sections - The Basics, Team Box Scores, Geography, Public Rankings, Play-by-play and Supplements. Please refer to the Data [[1]](#References) section at the bottom of this notebook for a full description of each file. On March 12, 2020, NCAA® canceled the Division I men's and women's 2020 basketball tournaments, as well as all remaining winter and spring NCAA® championships based on the evolving COVID-19 public health threat [[2]](#References), so the 2020 data is incomplete and does not have an information about 2020 NCAA® tournament bracket.\n\n\n## 1.2. Problem Statement\n\nThe goal of our project is to use data analysis to explain \"cinderellaness\" - **define common features and specific trends among \"Cinderella\" teams in NCAA® men's basketball**.\n\nThe intended solution is to:\n\n1. Define and filter out our target of interest - \"Cinderella\" teams\n2. Preprocess, filter and analyze input data\n3. Build visualizations to provide insights into the data & metrics\n4. Use machine learning to predict which team had the potential to become a Cinderella in a 2020 season\n5. Communicate the results of the analysis\n\n## 1.3. Methodology\n\n**Data Exploration and Preprocessing**\n\nScientific computing and analysis packages such as NumPy and Pandas will be used to **explore and preprocess the data**. Data cleaning will be performed where necessary. We will filter data across different categories, such as regular season vs. NCAA® tournament, all games vs. games won, team segment vs. metric vs. season.\n\n### Data Segmentation¶\n\nThe essential part of our analysis is to divide men's NCAA® basketball teams into 3 groups - **Cinderella**, **Top** and **Ordinary**.\n\nA March Madness Cinderella is a team that greatly exceeds its NCAA® tournament expectations. They are generally afterthoughts on the Selection Sunday bracket, but wind up becoming one of the biggest stories of the tournament [[3]](#References). In NCAA®, the field of teams is divided into four geographical regions. Each region has between 16 and 18 teams, which are assigned a seed number of one through 18, with the best team in the region awarded the No. 1 seed. Traditionally, the highest seeds (Nos. 1 through 8) have enjoyed more success than the lower seeds (Nos. 9 through 16). The lower seeds represent potential Cinderellas of the tournament. A Cinderella team is one that unexpectedly achieves success in the tournament. Traditionally, Cinderella's chariot turns back into a pumpkin before getting to the Final Four [[4]](#References) (also see Figure 1).\n\nConsidering the above definition, we decided to use the following segmentation as the foundation for our discussion and analysis:\n\n1. CINDERELLA - any basketball team **seeded 10th or worse that has advanced to the Round 3** in NCAA® tournament. This group is our target of interest.\n2. TOP - **top-seeded (Nos. 1 through 4**) teams who have advanced to the Round 3 in NCAA® tournament. This group will represent the most competitive teams - teams that match expectations.\n3. ORDINARY - all the other NCAA® tournament teams not falling under two previous categories, for example seeded No 2. but lost in the first-round game. Note that all seed Nos. 5 through 9 will always fall under this category."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"from IPython.display import Image\n\n# A copy of the plot that we will create later in the Implementation section:\ndisplay(Image(\"https://raw.githubusercontent.com/evanca/data-analysis_kaggle_march-madness-analytics-2020/master/img/06.png\"))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n---\n\nHere is the example segmentation result based on season 2019 data:\n\n**CINDERELLA TEAM OF 2018-19** \n\nTwelfth-seeded Oregon Ducks defeated Wisconsin 72-54 in a first-round game and beated UC Irvine 73-54 in a second-round game to advance to Sweet 16, where they lost 49-53 to a No. 1 seed Virginia, making this a classical example of a March Madness Cinderella story.\n\n
DUCKS
\n\n
\n\n**TOP TEAMS OF 2018-19** \n\nNotice how 16 teams were seeded Nos. 1 through 4, but only 14 are included in our TOP category, because 2 of the top-seeded teams (Kansas State Wildcats and Kansas Jayhawks) have not \nadvanced to the Round 3. \n\n\n\t
BLUE DEVILS
\n\t
SEMINOLES
\n\t
BULLDOGS
\n\t
COUGARS
\n \n\n\t
WILDCATS
\n\t
FIGHTING TIGERS
\n\t
RED RAIDERS
\n\t
SPARTANS
\n \n\n\t
TAR HEELS
\n\t
BOILERMAKERS
\n\t
VOLUNTEERS
\n\t
WOLVERINES
\n \n\n\t
CAVALIERS
\n\t
HOKIES
\n\n---\n\n**Exploratory Visualizations**\n\nData visualization libraries such as **Matplotlib, Seaborn and Plotly** will be used to create the exploratory visualizations. We will utilize different types of graphs, including but not limited to box plots, scatter plots and bar plots, to compare the features across different dimensions and see how features are distributed. Image graphs are called \"**Figure** ...\" and interactive graphs (that are responsive to mouse-over events) are called \"... **Interactive graph**\" for readers' convenience. \n\n**Statistical Analysis**\n\nWe will use descriptive statistics and measures of central tendency, such as **mean** (the average) and **median** (the middle value), to quantitatively describe and summarize our features of interest. Considering that Top category teams are expected to outperform two other categories in most cases, we want to focus more on **comparing Cinderella teams to Ordinary teams**, for example analyze some metric for Cinderellas vs. median value of the same metric for Ordinary teams.\n\n**Machine Learning**\n\nWe will use machine learning techniques to speculate which teams could become Cinderellas in season 2020 if the tournament would not be canceled. Our intention is to try out different classifiers and choose whichever performs best. While machine learning is not the main focus of our study, we might also want to apply some model refinement techniques to meet a certain threshold. We will use **Scikit-learn, XGBoost and Imbalanced-learn** modules to implement model training, evaluation and improvement.\n\n---\n🏀 CLICK HERE TO SKIP TO THE RESULTS 🏀"},{"metadata":{},"cell_type":"markdown","source":"---\n# II. Implementation\n\n## Exploratory Data Analysis\nThis section contains all the source code used in the analysis, including the code to create figures, interactive graphs and descriptive numbers."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"%matplotlib inline\n\n# Import packages:\n\nimport numpy as np \nimport pandas as pd \npd.set_option('mode.chained_assignment', None)\n\nimport matplotlib.pyplot as plt\nimport matplotlib.pylab as pylab\n\nimport seaborn as sns\n\nfrom plotly.subplots import make_subplots\nimport plotly.graph_objects as go\n\n# Define default seaborn plot params:\nsns.set(rc={'figure.figsize':(14,10)})\nsns.set_palette(\"colorblind\")\n\n# Define default matplotlib plot params:\nparams = {'figure.figsize':(14,10),\n 'figure.titlesize':16,\n 'axes.titlesize':'x-large',\n 'axes.labelsize':'large',\n 'xtick.labelsize':'large',\n 'ytick.labelsize':'large',\n 'legend.fontsize':'large'}\npylab.rcParams.update(params)\n\n# Define default plotly plot params:\nplotly_width = 880\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"file_nr = 1\n\ndef save_plot():\n '''Save plot into a ##.png file'''\n global file_nr\n if sys.executable != '/opt/conda/bin/python': # if running this notebook locally\n plt.savefig('kaggle/working/' + (str(file_nr).zfill(2)) + '.png', bbox_inches='tight', pad_inches=1)\n else:\n plt.savefig((str(file_nr).zfill(2)) + '.png', bbox_inches='tight', pad_inches=0.5)\n print(\"File nr. {}\".format(file_nr))\n file_nr +=1","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"import sys\n\nmen_dir = \"/kaggle/input/march-madness-analytics-2020/MDataFiles_Stage2/\"\n\nif sys.executable != '/opt/conda/bin/python':\n # remove the forward slash if running this notebook locally:\n men_dir = men_dir[1:]\n \ndef load_file(df, name):\n '''Load in the file and show basic info'''\n print(\"File: {}\".format(name))\n df = pd.read_csv(men_dir + name + '.csv')\n print(\"Num rows: {}\".format(len(df)))\n print(\"NaN values: {}\".format(df.isna().sum().sum()))\n print(\"Duplicated rows: {}\".format(df.duplicated().sum()))\n print(pd.concat([df.head(3), df.tail(2)]))\n return(df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2.1. The Basics, 1985-2020\n\n**Data Section 1 file: MRegularSeasonCompactResults.csv** - this file identifies the game-by-game results for many seasons of historical data, starting with the 1985 season (the first year the NCAA® had a 64-team tournament) [[1]](#References).\n\nWe will check each file that we load for a data quality issues like Null values and duplicate rows."},{"metadata":{"trusted":true},"cell_type":"code","source":"MRegularSeasonCompactResults = None\nMRegularSeasonCompactResults = load_file(MRegularSeasonCompactResults, \"MRegularSeasonCompactResults\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Data Section 1 file: MNCAATourneyCompactResults.csv** - this file identifies the game-by-game NCAA® tournament results for all seasons of historical data [[1]](#References)."},{"metadata":{"trusted":true},"cell_type":"code","source":"MNCAATourneyCompactResults = None\nMNCAATourneyCompactResults = load_file(MNCAATourneyCompactResults, \"MNCAATourneyCompactResults\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Calculate scoring margin** (a difference between the number of points scored by the winning team and the number of points scored by the losing team) for both dataframes:"},{"metadata":{"trusted":true},"cell_type":"code","source":"for df in [MRegularSeasonCompactResults, MNCAATourneyCompactResults]:\n df['Scoring margin'] = df['WScore'] - df['LScore']\n \nMRegularSeasonCompactResults.sample(3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"MNCAATourneyCompactResults.sample(3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Is it easier to win on the home court?\n\nSee how many games were won on each of the locations:"},{"metadata":{"trusted":true},"cell_type":"code","source":"MRegularSeasonCompactResults.WLoc.value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Total regular season games that were not on the neutral court:"},{"metadata":{"trusted":true},"cell_type":"code","source":"len(MRegularSeasonCompactResults[MRegularSeasonCompactResults.WLoc != \"N\"])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Create a plot. Note that we have added all figure numbers later, after we knew their order in the [Results](#II.-Results) section."},{"metadata":{"trusted":true},"cell_type":"code","source":"colors = [sns.color_palette(\"cubehelix\", 10)[6], sns.color_palette(\"cubehelix\", 10)[1], 'gold']\n\ndf = MRegularSeasonCompactResults[MRegularSeasonCompactResults.WLoc != \"N\"]\nprint(f'{df.Season.min()}-{df.Season.max()}')\n\nsns.scatterplot(x=\"LScore\", y=\"WScore\", data=df,\n hue=\"WLoc\", palette=colors[:-1], edgecolor=None, s=50, alpha=0.35)\n\nplt.xlabel(\"Points scored by the losing team\")\nplt.ylabel(\"Points scored by the winning team\")\n\nax = plt.gca()\nlegend = ax.legend()\nlegend.texts[0].set_text(\"Location\")\nlegend.texts[1].set_text(\"Home\")\nlegend.texts[2].set_text(\"Visiting\")\n\nplt.title('Figure 7. Points scored vs. home or visiting winner team,\\n 150K regular season games, 1985-2020.\\n')\n\nsave_plot()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Total regular season games in our data:"},{"metadata":{"trusted":true},"cell_type":"code","source":"len(MRegularSeasonCompactResults)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"### Plot 1 ###\n\ndf = MRegularSeasonCompactResults\nprint(f'{df.Season.min()}-{df.Season.max()}')\n\nsns.lineplot(x=\"Season\", y=\"Scoring margin\", data=df,\n hue=\"WLoc\", hue_order=['H', 'A', 'N'],\n palette=colors)\n\nplt.xlabel(\"Season\")\n\nax = plt.gca()\nlegend = ax.legend()\nlegend.texts[0].set_text(\"Location\")\nlegend.texts[1].set_text(\"Home\")\nlegend.texts[2].set_text(\"Visiting\")\nlegend.texts[3].set_text(\"Neutral\")\n\nplt.title('Figure 8. Scoring margin vs. winner team location (including neutral court games),\\n 167K regular season games, 1985-2020.\\n')\n\nsave_plot()\nplt.show()\n\n\n### Plot 2 ###\n\nplt.figure(figsize=(10,8))\n\nsns.scatterplot(x=\"Season\", y=\"Scoring margin\", data=MRegularSeasonCompactResults.sample(1000, random_state=0),\n hue=\"WLoc\", edgecolor='w', alpha=0.5, s=75, hue_order=['H', 'A', 'N'],\n palette=colors)\n\nplt.xlabel(\"Season\")\n\nax = plt.gca()\nlegend = ax.legend()\nlegend.texts[0].set_text(\"Location\")\nlegend.texts[1].set_text(\"Home\")\nlegend.texts[2].set_text(\"Visiting\")\nlegend.texts[3].set_text(\"Neutral\")\n\nplt.title('a closer look: random sample of 1000 games\\n')\n\nplt.show()\n\n\nprint(\"Descriptive statistics for file nr. {}:\".format(str(file_nr-1)))\nMRegularSeasonCompactResults[['WLoc', 'Scoring margin']].groupby('WLoc').describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The smaller plot (\"a closer look\") did not make it to the [Results](#III.-Results) section, but you can see how games on a visiting court almost never had a scoring margin above 30 (for a winning teams on this particular sample).\n___"},{"metadata":{},"cell_type":"markdown","source":"**Create a game round column for NCAA® tournaments** \n\nBecause of the consistent structure of the tournament schedule, we can actually tell what round a game was, depending on the exact DayNum [[1]](#References). Thus:\n\n- DayNum=134 or 135 (Tue/Wed) - play-in \n- DayNum=136 or 137 (Thu/Fri) - Round 1\n- DayNum=138 or 139 (Sat/Sun) - Round 2\n- DayNum=143 or 144 (Thu/Fri) - Round 3\n- DayNum=145 or 146 (Sat/Sun) - Round 4\n- DayNum=152 (Sat) - Round 5\n- DayNum=154 (Mon) - Round 6 (national final) "},{"metadata":{"trusted":true},"cell_type":"code","source":"MNCAATourneyCompactResults['Round'] = MNCAATourneyCompactResults['DayNum'] # copy DayNum column\nMNCAATourneyCompactResults['Round'].replace({134: \"Play-in\",\n 135: \"Play-in\",\n 136: \"Round 1\",\n 137: \"Round 1\",\n 138: \"Round 2\",\n 139: \"Round 2\",\n 143: \"Sweet 16\",\n 144: \"Sweet 16\",\n 145: \"Elite 8\",\n 146: \"Elite 8\",\n 152: \"Final 4\",\n 154: \"National Final\"}, inplace=True) # replace values with round names\n\n# Also add numerical round values for easier sorting:\nMNCAATourneyCompactResults['NumRound'] = MNCAATourneyCompactResults['DayNum'] # copy DayNum column\nMNCAATourneyCompactResults['NumRound'].replace({134: 0,\n 135: 0,\n 136: 1,\n 137: 1,\n 138: 2,\n 139: 2,\n 143: 3,\n 144: 3,\n 145: 4,\n 146: 4,\n 152: 5,\n 154: 6}, inplace=True) # replace values with round names\n\nMNCAATourneyCompactResults.sample(3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### What is the importance of seeding in March Madness?\n\nThe men’s college basketball tournament is made up of 68 teams. On Selection Sunday, before any tournament game is played, those teams are ranked 1 through 68 by the Selection Committee, with the best team in college basketball — based on regular season and conference tournament performance — sitting at No. 1. Four of those teams are eliminated in the opening round of the tournament (known as the First Four), leaving us with a field of 64 for the first round. Those 64 teams are split into four regions of 16 teams each, with each team being ranked 1 through 16. That ranking is the team’s seed [[23]](#References).\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"MNCAATourneySeeds = pd.read_csv(men_dir + '/MNCAATourneySeeds.csv')\nMNCAATourneySeeds.sample(5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Data segmentation - Cinderella, Top, Ordinary"},{"metadata":{},"cell_type":"markdown","source":"#### Create a numeric seed column"},{"metadata":{"trusted":true},"cell_type":"code","source":"MNCAATourneySeeds['SeedNo'] = MNCAATourneySeeds.Seed.str.extract('(\\d+)').astype(np.int64)\nMNCAATourneySeeds.sample(5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Connect seed numbers with the NCAA® tournament data\nMerge dataframes on season and winner team ID:"},{"metadata":{"trusted":true},"cell_type":"code","source":"len(MNCAATourneyCompactResults)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"MNCAATourneyCompactResults = pd.merge(MNCAATourneyCompactResults,\n MNCAATourneySeeds,\n how='inner',\n left_on=['Season', 'WTeamID'],\n right_on=['Season', 'TeamID'])\n\nMNCAATourneyCompactResults = MNCAATourneyCompactResults.drop('TeamID', 1)\n\nMNCAATourneyCompactResults","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"To filter out Cinderella teams we will look at any basketball team **seeded 10th or worse that has advanced to the Round 3**\n\nSeeded 10th or worse:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Seeded 10th or worse:\npossible_cinderellas = MNCAATourneyCompactResults[MNCAATourneyCompactResults['SeedNo'] >= 10]\npossible_cinderellas","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Advanced to the Round 3:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Round 2 is DayNum=138 or 139 (Sat/Sun), to bring the tournament field from 32 teams to 16 teams (to SWEET 16):\ncinderellas = possible_cinderellas[possible_cinderellas['DayNum'] >= 138] # played in Round 2\ncinderellas[\"Cinderella\"] = 1\ncinderellas = cinderellas[['Season', 'WTeamID', 'Cinderella']].drop_duplicates() # won in Round 2 (will play in Round 3)\ncinderellas","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Data Section 1 file: MTeams.csv** - this file identifies the different college teams present in the dataset. Each school is uniquely identified by a 4 digit id number [[1]](#References)."},{"metadata":{"trusted":true},"cell_type":"code","source":"MTeams = None\nMTeams = load_file(MTeams, \"MTeams\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### The resulting list of Cinderella teams"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Group by season and winner team id:\nseason_team_cinderellas = cinderellas.groupby(['Season','WTeamID'], as_index=False).mean()\nseason_team_cinderellas = season_team_cinderellas.sort_values(by='Season')\n\n# Print out the resulting list of cinderella teams:\nfor index, row in season_team_cinderellas.iterrows():\n team_id = season_team_cinderellas['WTeamID'][index]\n print(\"Season: {}; Team: {}\".format(season_team_cinderellas['Season'][index], MTeams.loc[MTeams['TeamID'] == team_id, 'TeamName'].values[0]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### How many Cinderella teams did each season have?"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(14,4))\n\ndf = season_team_cinderellas\nprint(f'{df.Season.min()}-{df.Season.max()}')\n\ng = sns.countplot(season_team_cinderellas.Season, palette=sns.color_palette(\"colorblind\")[1:2])\n\ng.set_xticklabels(g.get_xticklabels(), rotation=45)\nplt.ylabel(\"Cinderella teams\")\n\nplt.title(\"Figure 9. Cinderella team count per season,\\n1985-2019.\")\n\nsave_plot()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"A closer look at one example (2019, team Oregon):"},{"metadata":{"trusted":true},"cell_type":"code","source":"# See the 2019 example:\nMNCAATourneyCompactResults[((MNCAATourneyCompactResults['WTeamID'] == 1332) | (MNCAATourneyCompactResults['LTeamID'] == 1332))\n & (MNCAATourneyCompactResults['Season'] == 2019)]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From above table: team 1332 (Oregon) won in Round 1 and Round 2, and lost in the Sweet 16 to the team 1438.\n___"},{"metadata":{},"cell_type":"markdown","source":"**Make a separate group for the top-seeded teams** who has advanced to the Round 3.\n\nThis group will represent most competitive teams (high seed and high performance)."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Seeded 1, 2, 3 or 4:\ntop_seeded = MNCAATourneyCompactResults[MNCAATourneyCompactResults['SeedNo'] <= 4]\n\n# Round 2 is DayNum=138 or 139 (Sat/Sun), to bring the tournament field from 32 teams to 16 teams (to SWEET 16):\ntop_seeded = top_seeded[top_seeded['DayNum'] >= 138]\ntop_seeded[\"Top\"] = 1\ntop_seeded = top_seeded[['Season', 'WTeamID', 'Top']].drop_duplicates()\ntop_seeded","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"A closer look at 2019 season:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Group by season and team id in SEASON 2019 ONLY:\nseason_team_top_2019 = top_seeded[top_seeded[\"Season\"] == 2019].groupby(['Season','WTeamID'], as_index=False).mean()\n\n# Print out the resulting list of top teams in SEASON 2019 ONLY:\nprint(\"Season 2019 Top teams:\\n\")\nfor index, row in season_team_top_2019.iterrows():\n team_id = season_team_top_2019['WTeamID'][index]\n print(\"Season: {}; Team: {}\".format(season_team_top_2019['Season'][index], MTeams.loc[MTeams['TeamID'] == team_id, 'TeamName'].values[0]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Create a new dataframe to compare Cinderella teams vs. Top vs. Ordinary teams\nFilter by season - we don't want to include seasons without any cinderella teams:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Filter by season - we don't want to include seasons without any cinderella teams:\n\n### Regular season ###\nlabeled_MRegularSeasonCompactResults = MRegularSeasonCompactResults[MRegularSeasonCompactResults['Season'].isin(season_team_cinderellas['Season'].tolist())]\n\n### Tournaments ###\nlabeled_MNCAATourneyCompactResults = MNCAATourneyCompactResults[MNCAATourneyCompactResults['Season'].isin(season_team_cinderellas['Season'].tolist())]\nlabeled_MNCAATourneyCompactResults","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, **finish encoding labels**. Merge initial dataframes (regular season and tournament data) with our lists of Cinderella and Top teams (on season and winner team ID):"},{"metadata":{"trusted":true},"cell_type":"code","source":"### Regular season ###\nprint(len(labeled_MRegularSeasonCompactResults))\n\nlabeled_MRegularSeasonCompactResults = pd.merge(labeled_MRegularSeasonCompactResults,\n cinderellas,\n how='left',\n on=['Season', 'WTeamID'])\n\nlabeled_MRegularSeasonCompactResults = pd.merge(labeled_MRegularSeasonCompactResults,\n top_seeded,\n how='left',\n on=['Season', 'WTeamID'])\n\nlabeled_MRegularSeasonCompactResults","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"### Tournaments ###\nprint(len(labeled_MNCAATourneyCompactResults))\n\nlabeled_MNCAATourneyCompactResults = pd.merge(labeled_MNCAATourneyCompactResults,\n cinderellas,\n how='left',\n on=['Season', 'WTeamID'])\n\nlabeled_MNCAATourneyCompactResults = pd.merge(labeled_MNCAATourneyCompactResults,\n top_seeded,\n how='left',\n on=['Season', 'WTeamID'])\n\nlabeled_MNCAATourneyCompactResults","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Create a categorical LABEL column**"},{"metadata":{"trusted":true},"cell_type":"code","source":"### Regular season ###\n\n# Create a categorical LABEL column:\nlabel = labeled_MRegularSeasonCompactResults[['Cinderella', 'Top']]\nlabel = pd.DataFrame(label.idxmax(1))\nlabeled_MRegularSeasonCompactResults['LABEL'] = label\n\n# Fill in the missing values:\nlabeled_MRegularSeasonCompactResults['LABEL'] = labeled_MRegularSeasonCompactResults['LABEL'].fillna(\"Ordinary\")\n\nlabeled_MRegularSeasonCompactResults","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"### Tournaments ###\n\n# Create a categorical LABEL column:\nlabel = labeled_MNCAATourneyCompactResults[['Cinderella', 'Top']]\nlabel = pd.DataFrame(label.idxmax(1))\nlabeled_MNCAATourneyCompactResults['LABEL'] = label\n\n# Fill in the missing values:\nlabeled_MNCAATourneyCompactResults['LABEL'] = labeled_MNCAATourneyCompactResults['LABEL'].fillna(\"Ordinary\")\n\n# Sort value by round:\nlabeled_MNCAATourneyCompactResults = labeled_MNCAATourneyCompactResults.sort_values(by='NumRound', ascending=False) # 6, 5, 4...\nlabeled_MNCAATourneyCompactResults","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Check the results of data segmentation:"},{"metadata":{"trusted":true},"cell_type":"code","source":"### Regular season ###\nlabeled_MRegularSeasonCompactResults.LABEL.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"### Tournaments ###\nlabeled_MNCAATourneyCompactResults.LABEL.value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Fill in the missing values:"},{"metadata":{"trusted":true},"cell_type":"code","source":"### Regular season ###\n\n# Fill in the missing values:\nlabeled_MRegularSeasonCompactResults['Cinderella'] = labeled_MRegularSeasonCompactResults['Cinderella'].fillna(0) # not a cinderella\nlabeled_MRegularSeasonCompactResults['Top'] = labeled_MRegularSeasonCompactResults['Top'].fillna(0) # not a top\n\nlabeled_MRegularSeasonCompactResults","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"### Tournaments ###\n\n# Fill in the missing values:\nlabeled_MNCAATourneyCompactResults['Cinderella'] = labeled_MNCAATourneyCompactResults['Cinderella'].fillna(0) # not a cinderella\nlabeled_MNCAATourneyCompactResults['Top'] = labeled_MNCAATourneyCompactResults['Top'].fillna(0) # not a top\n\nlabeled_MNCAATourneyCompactResults","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Define label order and colors for future plots:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Label order in all plots:\norder=['Ordinary', 'Cinderella', 'Top']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Label colors in all plots:\nsns.palplot(sns.color_palette(\"colorblind\", 3))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Prepare a functions that will help us compare Cinderella teams vs. Ordinary teams:\n\ndef cinderella_vs_ordinary(df, games, season, metric_name):\n '''A function to print comparison of Cinderella team metric\n vs. Ordinary team median value of the same metric'''\n \n df_cinderella = df[df.Cinderella == 1.0]\n df_ordinary = df[df.LABEL == 'Ordinary']\n \n total_cinderella_games = len(df_cinderella)\n total_ordinary_games = len(df_ordinary)\n \n cinderella_mean = round(df_cinderella[metric_name].mean(), 2)\n ordinary_mean = round(df_ordinary[metric_name].mean(), 2)\n \n cinderella_median = round(df_cinderella[metric_name].median(), 2)\n ordinary_median = round(df_ordinary[metric_name].median(), 2)\n \n \n ### MORE THAN ORDINARY MEDIAN\n\n total_larger = len(df_cinderella[df_cinderella[metric_name] > \n ordinary_median])\n total_larger_ordinary = len(df_ordinary[df_ordinary[metric_name] > \n ordinary_median])\n \n share = total_larger/total_cinderella_games\n share_ordinary = total_larger_ordinary/total_ordinary_games\n \n def print_share_message(s='more'):\n '''Input string: \"more\" or \"less\"'''\n print(\"\\nIn {} of games {} in {}, Cinderella teams had {} than {} {} \"\n \"(mean: {}, median: {}) vs. {} of games \"\n \"for the Ordinary teams (mean: {}, median: {}).\".format(share_str, games, season, s, \n ordinary_median, metric_name,\n cinderella_mean, cinderella_median,\n share_ordinary_str, ordinary_mean, ordinary_median))\n \n if share > 0.51:\n share_str = '{:.0%}'.format(share)\n share_ordinary_str = '{:.0%}'.format(share_ordinary)\n \n print_share_message(\"more\")\n \n \n ### LESS THAN ORDINARY MEDIAN\n \n total_less = len(df_cinderella[df_cinderella[metric_name] < \n ordinary_median])\n total_less_ordinary = len(df_ordinary[df_ordinary[metric_name] < \n ordinary_median])\n \n share = total_less/total_cinderella_games\n share_ordinary = total_less_ordinary/total_ordinary_games\n \n if share > 0.52:\n share_str = '{:.0%}'.format(share)\n share_ordinary_str = '{:.0%}'.format(share_ordinary)\n\n print_share_message(\"less\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Do Cinderellas outscore the opponent by a wide margin?"},{"metadata":{},"cell_type":"markdown","source":"Plot scoring margin distribution vs. winner team category:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Define mean \"triangle\" marker for boxplots:\nmeanprops={\"markerfacecolor\":\"white\", \"markeredgecolor\":\"white\"} \n\ndf = labeled_MRegularSeasonCompactResults\nprint(f'{df.Season.min()}-{df.Season.max()}')\n\nfig, ax = plt.subplots(2,1, figsize = (14, 8), sharex=True)\n\nsns.boxplot(x='Scoring margin', y='LABEL', data=labeled_MRegularSeasonCompactResults, showmeans=True, ax=ax[0],\n order=order,\n orient='h',\n meanprops=meanprops, showfliers = False, width=0.5)\n\nsns.boxplot(x='Scoring margin', y='LABEL', data=labeled_MNCAATourneyCompactResults, showmeans=True, ax=ax[1],\n order=order,\n orient='h',\n meanprops=meanprops, showfliers = False, width=0.5)\n\nax[0].set_title('Regular season')\nax[0].set_xlabel(\"\")\nax[0].set_ylabel(\"\")\n\nax[1].set_title('Tournaments')\nax[1].set_ylabel(\"\")\n\nplt.suptitle(\"Figure 10. Scoring margin distribution vs. winner team category,\\n1985-2019.\", y = 1.05)\n\nsave_plot()\nplt.show()\n\n\nprint(\"Descriptive statistics for file nr. {}:\".format(str(file_nr-1)))\nprint('\\nRegular season')\nprint(labeled_MRegularSeasonCompactResults.groupby(['LABEL'])[\"Scoring margin\"].describe())\n\nprint('\\nTournaments')\nprint(labeled_MNCAATourneyCompactResults.groupby(['LABEL'])[\"Scoring margin\"].describe())\n\ncinderella_vs_ordinary(labeled_MRegularSeasonCompactResults, \"won\", \"regular season\", \"Scoring margin\")\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"___\n\n### How does the scoring margin change from round to round?"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Labels for round ticks:\ndf = labeled_MNCAATourneyCompactResults.groupby(['NumRound', 'Round'], as_index=False).count()[['Round']]\nlist(df['Round'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"g = sns.lineplot(x=\"NumRound\", y=\"Scoring margin\", hue=\"LABEL\", data=labeled_MNCAATourneyCompactResults,\n hue_order=order, ci=None)\n\nplt.title(\"Figure 11. Mean scoring margin vs. round and winner team category,\\ntournaments, 1985-2019.\\n\")\n\nax = plt.gca()\nhandles, labels = ax.get_legend_handles_labels()\nplt.legend(handles=handles[1:], labels=labels[1:])\n\nplt.xlabel(\"Round\")\n\ng.set_xticklabels([0] + list(df['Round']))\n\nsave_plot()\nplt.show()\n\nprint(\"Descriptive statistics for file nr. {}:\".format(str(file_nr-1)))\nlabeled_MNCAATourneyCompactResults.groupby(['NumRound','LABEL'])[\"Scoring margin\"].describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"---\nIn order to reward better teams, first-round matchups are determined by pitting the top team in the region against the bottom team (No. 1 vs. No. 16). Then the next highest vs. the next lowest (No. 2 vs. No. 15), and so on. In theory, this means that **the 1 seeds have the easiest opening matchup to win in the bracket** [[23]](#References).\n\n### How many teams of each category do we have per each seed no. and round?"},{"metadata":{"trusted":true},"cell_type":"code","source":"df = labeled_MNCAATourneyCompactResults[['SeedNo', 'Round', 'NumRound', 'WTeamID',\n 'LABEL']].groupby(['SeedNo', 'Round', 'NumRound',\n 'LABEL'], as_index=False).count()\n\ndf = df.sort_values(by='NumRound', ascending=False) # 6, 5, 4...\n\nsns.swarmplot(\"SeedNo\", \"Round\", hue=\"LABEL\", hue_order=order, data=df, size=15, palette=\"colorblind\")\n\nplt.xlim(0,17)\nplt.xticks(np.arange(1, 17, step=1))\n\nplt.xlabel(\"Seed nr.\")\n\nplt.legend(title=None)\n\nplt.title(\"Figure 1. Data segmentation. Presense of team category in each round\\nby seed number, 1985-2019 tournaments.\\n\")\n\nsave_plot()\nsns.despine()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Check font family (to use in fig.update_layout):\nplt.rcParams['font.family']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Count how many times this label won (per season):\ndf = labeled_MNCAATourneyCompactResults[['SeedNo', 'Round', 'NumRound', 'WTeamID',\n 'LABEL']].groupby(['SeedNo', 'Round', 'NumRound',\n 'LABEL'], as_index=False).count()\n\ndf = df.sort_values(by='NumRound') # 1, 2, 3...\n\n# Prepare hover text:\nhover_text = []\nfor index, row in df.iterrows():\n hover_text.append(('Seed no.: {SeedNo}
'+\n 'Team category: {LABEL}
'+\n 'Total games won: {WTeamID}').format(SeedNo=row['SeedNo'],\n LABEL=row['LABEL'],\n WTeamID=row['WTeamID']))\n\ndf['text'] = hover_text\n\n# Create figure\nfig = go.Figure()\n\ni = 0\nfor label in order:\n plot_df = df[df.LABEL == label]\n size = plot_df['WTeamID']\n fig.add_trace(go.Scatter(\n x=plot_df['SeedNo'], y=plot_df['Round'],\n mode='markers',\n text=plot_df['text'],\n name=label,\n marker_size=plot_df['WTeamID'],\n marker=dict(\n size=size,\n sizemode='area',\n sizeref=0.18, # setting 'sizeref' to less than 1 increases marker sizes\n sizemin=2,\n line_width=3,\n line_color=sns.color_palette(\"colorblind\").as_hex()[i]), # outline color\n marker_color='rgba(0, 0, 0, 0)' # inside color\n ))\n i+=1\n \n# Move legend:\nfig.update_layout(legend=dict(x=0.835, y=0.95, bgcolor='rgba(0, 0, 0, 0)'))\n\n\n# Add titles:\n\nfig.update_xaxes(title_text='Seed no.')\nfig.update_yaxes(title_text='Round')\n\n# Improve tick frequency:\nfig.update_layout(xaxis = dict(tickmode = 'array', tickvals = list(range(1, 17))))\n\n# Set size:\nfig.update_layout(width=plotly_width, height=650)\n\n# Set axis font:\nfig.update_yaxes(tickfont=dict(size=14))\n\n# Plot title:\nfig.update_layout(\n title={\n 'text': \"Games won by team category and seed number,
1985-2019 tournaments. Interactive graph.\",\n 'y':0.95,\n 'x':0.5,\n 'xanchor': 'center',\n 'yanchor': 'top'},\n font=dict(\n family='sans-serif',\n color=\"#000\"))\n\n\nfig.show(renderer=\"kaggle\")\n\nfig_2 = go.Figure(fig) # to show the same fig in the Results section\n\n\nprint(\"Total games won per round:\")\ndf.groupby(['NumRound','LABEL'])['WTeamID'].sum()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### How often have Cinderellas won in overtime?"},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.swarmplot(x=\"NumOT\", y='WTeamID', hue=\"LABEL\", hue_order=order,\n data=labeled_MNCAATourneyCompactResults,\n alpha=0.75,\n s=8)\n\nplt.legend(title=None, bbox_to_anchor=(0.575, 1), loc=2)\n\nplt.xlabel(\"Number of overtime periods in the game\")\n\nax = plt.gca()\nax.get_yaxis().set_visible(False)\n\nplt.title(\"Games won vs. number of overtime periods in the game,\\ntournaments, 1985-2019.\\n\")\n\nplt.show()\n\nlabeled_MNCAATourneyCompactResults.groupby(['NumOT', 'LABEL'])['WTeamID'].count()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"*Note.* We decided to not include this figure to the [Results](#III.-Results) section, because we found it not especially insightful in terms of \"Cinderellaness\"."},{"metadata":{},"cell_type":"markdown","source":"## 2.2. Geography, 2010-2020\n\n**Data Section 3 file: Cities.csv** - this file provides a master list of cities that have been locations for games played [[1]](#References)."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Locations for games played\nCities = None\nCities = load_file(Cities, \"Cities\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"Cities.State.value_counts().nunique() # how many unique states do we have?","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Data Section 3 file: MGameCities.csv** - this file identifies all games, starting with the 2010 season, along with the city that the game was played in. Games from the regular season, the NCAA® tourney, and other post-season tournaments, are all listed together [[1]](#References). "},{"metadata":{"trusted":true},"cell_type":"code","source":"MGameCities = None\nMGameCities = load_file(MGameCities, \"MGameCities\")\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Merge both geography data files together:"},{"metadata":{"trusted":true},"cell_type":"code","source":"new_MGameCities = pd.merge(MGameCities, Cities, on=['CityID'])\nassert len(new_MGameCities) == len(MGameCities), \"Wrong item count.\"\n\nnew_MGameCities","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Load the file with each city geo location [[24]](#References):"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Load the file with each city geo location:\n\ngeo_file = \"/kaggle/input/ncaageocities/geo_Cities.csv\"\n\nif sys.executable != '/opt/conda/bin/python':\n # remove the forward slash if running this notebook locally:\n geo_file = geo_file[1:] \n \ngeo_Cities = pd.read_csv(geo_file)\ngeo_Cities.sample(5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Join our new dataframe with the geo data:"},{"metadata":{"trusted":true},"cell_type":"code","source":"cols = ['CityID', 'City', 'State']\ngeo_MGameCities = new_MGameCities.join(geo_Cities.set_index(cols), on=cols)\n\nassert len(new_MGameCities) == len(geo_MGameCities), \"Wrong item count.\"\n\ngeo_MGameCities","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Connect regular game cities with regular season game-by-game results"},{"metadata":{"trusted":true},"cell_type":"code","source":"geo_MRegularSeasonCompactResults = pd.merge(geo_MGameCities[geo_MGameCities['CRType'] == 'Regular'],\n MRegularSeasonCompactResults,\n how='inner',\n on=['Season', 'DayNum', 'WTeamID', 'LTeamID'],\n validate=\"one_to_one\")\n\ngeo_MRegularSeasonCompactResults","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Figure out team home location\n\nGame location should be \"H\":"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Group by 'WTeamID', mean:\nteam_homes = geo_MRegularSeasonCompactResults[geo_MRegularSeasonCompactResults[\"WLoc\"] == \"H\"][['WTeamID',\n 'CityID']].groupby(['WTeamID'], as_index = False).mean()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Testing out that each team has one unique home city:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Testing out that each team has one unique home city\nnp.array_equal(team_homes.CityID, team_homes.CityID.astype(int)) # output should be True","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Check what went wrong in the above code:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Checking what went wrong in the above code:\n\nteam_homes = geo_MRegularSeasonCompactResults[geo_MRegularSeasonCompactResults[\"WLoc\"] == \"H\"][['WTeamID','CityID', 'Season']].groupby(['WTeamID','CityID'], as_index = False).mean()\nteam_homes[team_homes['WTeamID'].duplicated(keep=False)]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"team_homes[team_homes['WTeamID'] == 1437]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The output shows that some teams have different home cities. Let's look at one specific example to further investigate this:"},{"metadata":{"trusted":true},"cell_type":"code","source":"MTeams[MTeams.TeamID == 1437]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"Cities[Cities.CityID.isin([4266, 4361, 4467])]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This example confirms that **sometimes one team has different home locations in the game-by-game data file**. These three cities (Philadelphia, Villanova, Bryn Mawr) are close to each other, so this is not an error but rather a data specific issue.\n\nTo avoid this kind of inconsistency, we will drop the duplicates. Note that we didn't investigate which of the cities is bigger or more significant for each team, here we only care about keeping one city per team."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Keep the home city from the latest season:\nteam_homes = team_homes.drop_duplicates('WTeamID')\n\nteam_homes[team_homes['WTeamID'] == 1437] # should be only one row","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"Cities[Cities.CityID == 4266]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Drop Season column, it makes no sense after applying mean. Also rename the columns:"},{"metadata":{"trusted":true},"cell_type":"code","source":"team_homes = team_homes.drop('Season', 1)\nteam_homes = team_homes.rename(columns={\"WTeamID\": \"TeamID\",\n \"CityID\": \"HomeCityID\"}) # rename columns\nteam_homes","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Add the geo location data:"},{"metadata":{"trusted":true},"cell_type":"code","source":"team_homes = pd.merge(team_homes, geo_Cities, left_on='HomeCityID', right_on='CityID', how='left')\nteam_homes = team_homes.drop('CityID', 1)\nteam_homes","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Count teams per each home town**"},{"metadata":{"trusted":true},"cell_type":"code","source":"team_homes_cnt = team_homes.groupby(['HomeCityID', 'City', 'State', 'Latitude', 'Longitude'], as_index=False).count()\n# Sort values for the bigger points to show above the small points:\nteam_homes_cnt = team_homes_cnt.sort_values(by='TeamID')\nteam_homes_cnt","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Just a test, team Michigan St should be [from East Lansing](https://www.ncaa.com/schools/michigan-st):"},{"metadata":{"trusted":true},"cell_type":"code","source":"team_homes[team_homes['TeamID'] == 1277]","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true},"cell_type":"code","source":"team_homes_cnt.TeamID.value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Which city is the \"capital\" of men's college basketball?"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# This module allows to avoid overlapping text on scatter plots:\n# Credit: https://github.com/Phlya/adjustText (The MIT License)\n!pip install adjustText\n\n# Parse SVG paths into matplotlib Path objects for plotting:\n# Credit: https://github.com/nvictus/svgpath2mpl (The 3-Clause BSD License)\n!pip install svgpath2mpl matplotlib","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nos.environ['PROJ_LIB'] = 'C:\\\\Users\\\\Ivanna\\\\Anaconda3\\\\pkgs\\\\basemap-1.2.0-py37h4e5d7af_0\\\\Lib\\\\site-packages\\\\mpl_toolkits\\\\basemap\\\\data\\\\'\n\n\nfrom mpl_toolkits.basemap import Basemap\nfrom adjustText import adjust_text\nfrom svgpath2mpl import parse_path\nimport matplotlib.patheffects as path_effects\nfrom matplotlib import cm\nfrom matplotlib.colors import ListedColormap, LinearSegmentedColormap\n\ndf = geo_MRegularSeasonCompactResults\nprint(f'{df.Season.min()}-{df.Season.max()}')\n\n# Create US map:\nmap = Basemap(llcrnrlon=-119,llcrnrlat=22,urcrnrlon=-64,urcrnrlat=49, fix_aspect=False)\n\nmap.drawmapboundary(fill_color='#cee2ee', linewidth=0)\nmap.fillcontinents(color='#fbf7f4')\n\nmap.drawcountries(linewidth=0.25)\nmap.drawcoastlines(linewidth=0.25)\n\nmap.drawstates(color='0.5', linewidth=0.25) # draw the American state border\n\n\n# Create custom marker:\nball = parse_path(\"\"\"M297,148.5C297,66.617,230.383,0,148.5,0S0,66.617,0,148.5S66.617,297,148.5,297S297,230.383,297,148.5z M211.044,156.5\nh-54.877v124.252c-2,0.158-5.314,0.248-8,0.248c-2.687,0-5-0.09-8-0.248V156.5H85.956c-1.665,31.936-13.236,61.29-31.687,85.051\nc-3.826-3.874-7.413-7.982-10.743-12.3c15.244-20.59,24.815-45.614,26.398-72.751H16.249c-0.159-2.648-0.249-5.314-0.249-8\ns0.09-5.352,0.249-8h53.676c-1.582-27.137-11.154-52.162-26.397-72.751c3.329-4.318,6.917-8.427,10.742-12.3\nC72.72,79.21,84.292,108.563,85.956,140.5h54.211V16.248c3-0.158,5.313-0.248,8-0.248c2.686,0,6,0.09,8,0.248V140.5h54.877\nc1.664-31.937,13.236-61.29,31.687-85.051c3.825,3.873,7.413,7.981,10.742,12.3c-15.243,20.589-24.815,45.614-26.397,72.751h53.676\nc0.159,2.648,0.249,5.314,0.249,8s-0.09,5.352-0.249,8h-53.676c1.583,27.137,11.154,52.161,26.398,72.751\nc-3.33,4.317-6.917,8.426-10.743,12.3C224.28,217.79,212.709,188.436,211.044,156.5z\"\"\")\n\n# Create a custom cmap based on a 'YlOrBr':\nYlOrBr = cm.get_cmap('YlOrBr', 100)\nnewcmp = ListedColormap(YlOrBr(np.linspace(0.3, 1, 256)))\n\n# Plot data on a map:\nmap.scatter(team_homes_cnt['Longitude'], # longitude goes first\n team_homes_cnt['Latitude'], # latitude goes second\n s=pow(team_homes_cnt['TeamID']*50, 1.5), # marker size\n c=team_homes_cnt['TeamID'], # marker color\n marker=ball,\n alpha=0.8,\n zorder=10,\n cmap=newcmp)\n\n\n# Annotate biggest points:\n\ntop = team_homes_cnt[team_homes_cnt['TeamID'] >= 3]\n\ntop_texts = [plt.text(top['Longitude'][i]+0.5,\n top['Latitude'][i]-0.5,\n top['City'][i],\n zorder=11) for i in top.index]\n\n# Add white outline to text:\nfor text in top_texts:\n text.set_path_effects([path_effects.Stroke(linewidth=3, foreground='white', alpha=.8),\n path_effects.Normal()])\n\n# Fix overlapping text:\nadjust_text(top_texts)\n\nplt.title(\"Figure 2. Top US cities by NCAA® men's basketball team count,\\nregular season, 2010-2020.\\n\")\n\nsave_plot()\nplt.show()\n\n\nteam_homes_cnt[team_homes_cnt['TeamID'] > 1].tail(10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Print out the team names for Philadelphia:"},{"metadata":{"trusted":true},"cell_type":"code","source":"for index, row in team_homes[team_homes['City'] == \"Philadelphia\"].iterrows():\n team_id = team_homes[team_homes['City'] == \"Philadelphia\"]['TeamID'][index]\n print(MTeams.loc[MTeams['TeamID'] == team_id, 'TeamName'].values[0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2.3. Team Box Scores, 2003-2019\n\n**Data Section 2 file: MNCAATourneyDetailedResults.csv** - this file provides team-level box scores for many NCAA® tournaments, starting with the 2003 season [[1]](#References)."},{"metadata":{"trusted":true},"cell_type":"code","source":"MNCAATourneyDetailedResults = None\nMNCAATourneyDetailedResults = load_file(MNCAATourneyDetailedResults, 'MNCAATourneyDetailedResults')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We will be using the below dictionary to look up the index of columns:"},{"metadata":{"trusted":true},"cell_type":"code","source":"print({c: i for i, c in enumerate(MNCAATourneyDetailedResults.columns)})","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Make each row in our dataframe about either a winning or a losing team stats**"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Columns about winning team:\n\nwinning = pd.concat([MNCAATourneyDetailedResults.iloc[:,:4], # game ID columns\n MNCAATourneyDetailedResults.iloc[:,4:5], # LTeamID\n MNCAATourneyDetailedResults.iloc[:,8:21], # WFGM, WFGA, WFGM3 ...\n MNCAATourneyDetailedResults.iloc[:,27:29]], # opponent OR, DR\n axis=1, sort=False)\n\nwinning['TeamID'] = winning['WTeamID']\nwinning['won'] = 1\n\nwinning # 'Season', 'DayNum', 'WTeamID', 'WScore', 'LTeamID'...","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Columns about losing team:\n\nlosing = pd.concat([MNCAATourneyDetailedResults.iloc[:,:3],\n MNCAATourneyDetailedResults.iloc[:,5:6], # LScore\n MNCAATourneyDetailedResults.iloc[:,4:5], # LTeamID\n MNCAATourneyDetailedResults.iloc[:,21:34],\n MNCAATourneyDetailedResults.iloc[:,14:16]], # opponent OR, DR\n axis=1, sort=False)\nlosing['TeamID'] = losing['LTeamID']\nlosing['won'] = 0\n\nlosing # 'Season', 'DayNum', 'WTeamID', 'LScore', 'LTeamID'...","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(list(winning))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(list(losing))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Resulting dataframe object will have a \"double_\" prefix, because each game is now represented twice - one row for a winning team and one row for a losing team:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Remove \"W\" and \"L\" prefixes:\n\nnew_columns = ['Season', 'DayNum', 'WTeamID', 'Score', 'LTeamID', # changed only \"Score\" here\n 'FGM', 'FGA', 'FGM3', 'FGA3', 'FTM', 'FTA', 'OR', 'DR', 'Ast', 'TO', 'Stl', 'Blk', 'PF', \n 'OppOR', 'OppDR', 'TeamID', 'won']\n\n# Rename columns:\nwinning.columns = new_columns\nlosing.columns = new_columns\n\n# Concatenate:\nframes = [winning, losing]\ndouble_MNCAATourneyDetailedResults = pd.concat(frames)\n\nassert(len(double_MNCAATourneyDetailedResults) == (len(winning) + len(losing)))\n\ndouble_MNCAATourneyDetailedResults ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Calculate Rebound Margin**\n\nRebound Margin = RPG - OPP RPG [[15]](#References)\n\nIf a team won, opponent is LTeamID, otherwise opponent is WTeamID. Following this logic we already created two columns with opponent rebounds - \"OppOR\" and \"OppDR\".\n\nTotal rebounds per game = offensive rebounds + defensive rebounds:"},{"metadata":{"trusted":true},"cell_type":"code","source":"double_MNCAATourneyDetailedResults['Rebound Margin'] = (double_MNCAATourneyDetailedResults['OR'] +\n double_MNCAATourneyDetailedResults['DR']) - \\\n (double_MNCAATourneyDetailedResults['OppOR'] +\n double_MNCAATourneyDetailedResults['OppDR'])\n \ndouble_MNCAATourneyDetailedResults.sample(3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Add categorical round information:"},{"metadata":{"trusted":true},"cell_type":"code","source":"double_MNCAATourneyDetailedResults['Round'] = double_MNCAATourneyDetailedResults['DayNum'] # copy DayNum column\ndouble_MNCAATourneyDetailedResults['Round'].replace({134: \"Play-in\",\n 135: \"Play-in\",\n 136: \"Round 1\",\n 137: \"Round 1\",\n 138: \"Round 2\",\n 139: \"Round 2\",\n 143: \"Sweet 16\",\n 144: \"Sweet 16\",\n 145: \"Elite 8\",\n 146: \"Elite 8\",\n 152: \"Final 4\",\n 154: \"National Final\"}, inplace=True) # replace values with round names\n\n# Also add numerical round values for easier sorting:\ndouble_MNCAATourneyDetailedResults['NumRound'] = double_MNCAATourneyDetailedResults['DayNum'] # copy DayNum column\ndouble_MNCAATourneyDetailedResults['NumRound'].replace({134: 0,\n 135: 0,\n 136: 1,\n 137: 1,\n 138: 2,\n 139: 2,\n 143: 3,\n 144: 3,\n 145: 4,\n 146: 4,\n 152: 5,\n 154: 6}, inplace=True) # replace values with round names\n\ndouble_MNCAATourneyDetailedResults['Round'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Calculate two-point field goals attempted\nFGA2 = FGA - FGA3"},{"metadata":{"trusted":true},"cell_type":"code","source":"double_MNCAATourneyDetailedResults['FGA2'] = double_MNCAATourneyDetailedResults['FGA'] - double_MNCAATourneyDetailedResults['FGA3']\ndouble_MNCAATourneyDetailedResults.sample(3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Calculate two-point field goals made\nFGM2 = FGM - FGM3"},{"metadata":{"trusted":true},"cell_type":"code","source":"double_MNCAATourneyDetailedResults['FGM2'] = double_MNCAATourneyDetailedResults['FGM'] - double_MNCAATourneyDetailedResults['FGM3']\ndouble_MNCAATourneyDetailedResults.sample(3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Add the Cinderella and Top labels defined earlier"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Filter by season - we don't want to include seasons without any cinderella teams:\nlabeled_double_MNCAATourneyDetailedResults = double_MNCAATourneyDetailedResults[double_MNCAATourneyDetailedResults['Season'].isin(season_team_cinderellas['Season'].tolist())]\n\ncinderellas = cinderellas.rename(columns={\"WTeamID\": \"TeamID\"}) # rename columns\ntop_seeded = top_seeded.rename(columns={\"WTeamID\": \"TeamID\"}) # rename columns\n\ncols = ['Season', 'TeamID']\nlabeled_double_MNCAATourneyDetailedResults = labeled_double_MNCAATourneyDetailedResults.join(cinderellas.set_index(cols), on=cols)\nlabeled_double_MNCAATourneyDetailedResults = labeled_double_MNCAATourneyDetailedResults.join(top_seeded.set_index(cols), on=cols)\n\nlabeled_double_MNCAATourneyDetailedResults","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"How many Cinderella team games do we have in this data:"},{"metadata":{"trusted":true},"cell_type":"code","source":"labeled_double_MNCAATourneyDetailedResults.Cinderella.value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Continue adding labels:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create a categorical LABEL column:\nlabel = labeled_double_MNCAATourneyDetailedResults[['Cinderella', 'Top']]\nlabel = pd.DataFrame(label.idxmax(1))\nlabeled_double_MNCAATourneyDetailedResults['LABEL'] = label\n\n# Fill in the missing values:\nlabeled_double_MNCAATourneyDetailedResults['LABEL'] = labeled_double_MNCAATourneyDetailedResults['LABEL'].fillna(\"Ordinary\")\n\n# Fill in the missing values:\nlabeled_double_MNCAATourneyDetailedResults['Cinderella'] = labeled_double_MNCAATourneyDetailedResults['Cinderella'].fillna(0) # not a cinderella\nlabeled_double_MNCAATourneyDetailedResults['Top'] = labeled_double_MNCAATourneyDetailedResults['Top'].fillna(0) # not a top\n\nlabeled_double_MNCAATourneyDetailedResults","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"labeled_double_MNCAATourneyDetailedResults.Cinderella.value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Data Section 2 file: MRegularSeasonDetailedResults.csv** - this file provides team-level box scores for many regular seasons of historical data, starting with the 2003 season [[1]](#References)."},{"metadata":{"trusted":true},"cell_type":"code","source":"MRegularSeasonDetailedResults = None\nMRegularSeasonDetailedResults = load_file(MRegularSeasonDetailedResults, 'MRegularSeasonDetailedResults')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Repeat all the same steps for the regular season detailed results**"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Columns about winning team:\n\nwinning = pd.concat([MRegularSeasonDetailedResults.iloc[:,:4],\n MRegularSeasonDetailedResults.iloc[:,4:5], # LTeamID\n MRegularSeasonDetailedResults.iloc[:,8:21],\n MRegularSeasonDetailedResults.iloc[:,27:29]], # opponent OR, DR\n axis=1, sort=False)\n\nwinning['TeamID'] = winning['WTeamID']\nwinning['won'] = 1\n\n# Columns about losing team:\n\nlosing = pd.concat([MRegularSeasonDetailedResults.iloc[:,:3],\n MRegularSeasonDetailedResults.iloc[:,5:6], # LScore\n MRegularSeasonDetailedResults.iloc[:,4:5], # LTeamID\n MRegularSeasonDetailedResults.iloc[:,21:34],\n MRegularSeasonDetailedResults.iloc[:,14:16]], # opponent OR, DR\n axis=1, sort=False)\n\nlosing['TeamID'] = losing['LTeamID']\nlosing['won'] = 0\n\n# Rename columns:\nwinning.columns = new_columns\nlosing.columns = new_columns\n\n# Concatenate:\nframes = [winning, losing]\ndouble_MRegularSeasonDetailedResults = pd.concat(frames)\n\nprint(len(double_MRegularSeasonDetailedResults))\n\ndouble_MRegularSeasonDetailedResults['Round'] = \"Regular Season\"\n\ndouble_MRegularSeasonDetailedResults['FGA2'] = double_MRegularSeasonDetailedResults['FGA'] - double_MRegularSeasonDetailedResults['FGA3']\ndouble_MRegularSeasonDetailedResults['FGM2'] = double_MRegularSeasonDetailedResults['FGM'] - double_MRegularSeasonDetailedResults['FGM3']\n\ndouble_MRegularSeasonDetailedResults\n\n# Filter by season - we don't want to include seasons without any cinderella teams:\n\nlabeled_double_MRegularSeasonDetailedResults = double_MRegularSeasonDetailedResults[double_MRegularSeasonDetailedResults['Season'].isin(season_team_cinderellas['Season'].tolist())]\n\ncols = ['Season', 'TeamID']\nlabeled_double_MRegularSeasonDetailedResults = labeled_double_MRegularSeasonDetailedResults.join(cinderellas.set_index(cols), on=cols)\nlabeled_double_MRegularSeasonDetailedResults = labeled_double_MRegularSeasonDetailedResults.join(top_seeded.set_index(cols), on=cols)\n\n# Create a categorical LABEL column:\nlabel = labeled_double_MRegularSeasonDetailedResults[['Cinderella', 'Top']]\nlabel = pd.DataFrame(label.idxmax(1))\nlabeled_double_MRegularSeasonDetailedResults['LABEL'] = label\n\n# Fill in the missing values:\nlabeled_double_MRegularSeasonDetailedResults['LABEL'] = labeled_double_MRegularSeasonDetailedResults['LABEL'].fillna(\"Ordinary\")\n\n# Fill in the missing values:\nlabeled_double_MRegularSeasonDetailedResults['Cinderella'] = labeled_double_MRegularSeasonDetailedResults['Cinderella'].fillna(0) # not a cinderella\nlabeled_double_MRegularSeasonDetailedResults['Top'] = labeled_double_MRegularSeasonDetailedResults['Top'].fillna(0) # not a top\n\n# Calculate Rebound Margin:\nlabeled_double_MRegularSeasonDetailedResults['Rebound Margin'] = (labeled_double_MRegularSeasonDetailedResults['OR'] +\n labeled_double_MRegularSeasonDetailedResults['DR']) - \\\n (labeled_double_MRegularSeasonDetailedResults['OppOR'] +\n labeled_double_MRegularSeasonDetailedResults['OppDR'])\n\nlabeled_double_MRegularSeasonDetailedResults","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Check the resulting labels:"},{"metadata":{"trusted":true},"cell_type":"code","source":"labeled_double_MRegularSeasonDetailedResults.LABEL.value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Copy stats of winning teams to a separate dataframe** (for both regular season and tournaments)"},{"metadata":{"trusted":true},"cell_type":"code","source":"### Regular season ###\n\nreg_winning_stats = labeled_double_MRegularSeasonDetailedResults[labeled_double_MRegularSeasonDetailedResults['won'] == 1]\nreg_winning_stats.sample(3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"### Tournaments ###\n\ntourney_winning_stats = labeled_double_MNCAATourneyDetailedResults[labeled_double_MNCAATourneyDetailedResults['won'] == 1]\ntourney_winning_stats.sample(3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Make two lists with dataframes that will be used in plots:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Make two lists with dataframes that will be used in plots:\n\ndetailed_results_dfs = [labeled_double_MRegularSeasonDetailedResults,\n labeled_double_MNCAATourneyDetailedResults] # all games \n\nwinning_dfs = [reg_winning_stats, tourney_winning_stats] # games won","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def print_distribution_comments(metric_name):\n '''Print Cinderella vs. Ordinary stats for different seasons and games'''\n cinderella_vs_ordinary(detailed_results_dfs[0], \"played\", \"regular season\", metric_name)\n cinderella_vs_ordinary(detailed_results_dfs[1], \"played\", \"tournaments\", metric_name)\n cinderella_vs_ordinary(winning_dfs[0], \"won\", \"regular season\", metric_name)\n cinderella_vs_ordinary(winning_dfs[1], \"won\", \"tournaments\", metric_name)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Are Cinderellas a good shooters?"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = make_subplots(rows=2, cols=1,\n shared_xaxes=True, vertical_spacing = 0.15)\n\ntwo_point_x = [] # default data, all X values\nthree_point_x = [] # data to use on button click, all X values\n\nrow = 1 # row nr. for subplot\nfor df, won_df in zip(detailed_results_dfs, winning_dfs): # Make plots for both regular season and tournaments:\n \n i = 0\n for label in order: # 'Ordinary', 'Cinderella', 'Top'\n \n plot_df = df[df.LABEL == label] # all games \n won_plot_df = won_df[won_df.LABEL == label] # games won\n \n # All games (visible):\n fig.add_trace(\n go.Box(x=plot_df['FGM2'],\n name=label,\n marker_color=sns.color_palette(\"colorblind\").as_hex()[i],\n boxmean=True, # represent mean\n boxpoints='suspectedoutliers',\n visible=True), # only suspected outliers\n row=row, col=1)\n \n two_point_x.append(plot_df['FGM2'])\n three_point_x.append(plot_df['FGM3'])\n \n # Games won (not visible by default):\n fig.add_trace(\n go.Box(x=won_plot_df['FGM2'],\n name=label,\n marker_color=sns.color_palette(\"colorblind\").as_hex()[i],\n boxmean=True, # represent mean\n boxpoints='suspectedoutliers',\n visible=False), # only suspected outliers\n row=row, col=1)\n \n two_point_x.append(won_plot_df['FGM2'])\n three_point_x.append(won_plot_df['FGM3'])\n \n i+=1\n \n row+=1 # go to next subplot\n\n\n# Default visibility:\nshow_all_games = [True, False, True, False, True, False, # row 1: OO CC TT ('Ordinary', 'Cinderella', 'Top')\n True, False, True, False, True, False] # row 2: OO CC TT ('Ordinary', 'Cinderella', 'Top')\n# Opposite visibility (reverse above list):\nshow_games_won = show_all_games[::-1]\n\n \nfig.update_layout(showlegend=False, # hide ledend\n width=plotly_width, height=750) # set size\n\n# Set axis font:\nfig.update_yaxes(tickfont=dict(size=14))\n\ndefault_title = \"Two-point field goals made (distribution) vs. team category,
2003-2019. Interactive graph.\"\nhidden_title = \"Three-point field goals made (distribution) vs. team category,
2003-2019. Interactive graph.\"\n\ndefault_xtitle = dict(x=0.5, y=-0.1, xref=\"paper\", yref=\"paper\",\n text=\"Two-point field goals per game\",\n showarrow=False, font=dict(size=14))\nhidden_xtitle = dict(x=0.5, y=-0.1, xref=\"paper\", yref=\"paper\",\n text=\"Three-point field goals per game\",\n showarrow=False, font=dict(size=14))\n\nupper_subplot_title = dict(x=0.5, y=1.05, xref=\"paper\", yref=\"paper\",\n text=\"Regular season\", showarrow=False, font=dict(size=16))\nlower_subplot_title = dict(x=0.5, y=0.45, xref=\"paper\", yref=\"paper\",\n text=\"Tournaments\", showarrow=False, font=dict(size=16))\n\n# Add subplot titles:\nfig.add_annotation(upper_subplot_title)\nfig.add_annotation(lower_subplot_title)\n\n# Add annotations:\nfig.add_annotation(default_xtitle)\n\ndefault_annotations = [default_xtitle, upper_subplot_title, lower_subplot_title]\nhidden_annotations = [hidden_xtitle, upper_subplot_title, lower_subplot_title]\n\n# Add buttons:\nfig.update_layout(\n updatemenus=[\n dict( # these buttons will change data\n type=\"buttons\",\n direction=\"right\",\n active=0,\n x=0.45,\n y=1.2,\n buttons=list([\n dict(label=\"2-point goals\",\n method=\"update\",args=[{\"x\": two_point_x},\n {\"title\": default_title,\n \"annotations\": default_annotations},\n {\"visible\": show_all_games}]), \n dict(label=\"3-point goals\",\n method=\"update\",args=[{\"x\": three_point_x},\n {\"title\": hidden_title,\n \"annotations\": hidden_annotations},\n {\"visible\": show_all_games}])\n ]),\n ),\n dict( # these buttons will change visibility of \"games won\"\n buttons=list([\n dict(label=\"All games\",\n method=\"restyle\",args=[{\"visible\": show_all_games}]), \n dict(label=\"Games won\",\n method=\"restyle\",args=[{\"visible\": show_games_won}])\n ]),\n direction=\"down\",\n showactive=True,\n x=0.8,\n y=1.2,\n )\n ])\n\n# Plot title:\nfig.update_layout(\n title={\n 'text': default_title,\n 'y':0.95,\n 'x':0.5,\n 'xanchor': 'center',\n 'yanchor': 'top'},\n font=dict(\n family='sans-serif',\n color=\"#000\"),\n margin=dict(t=180) # margin between title and plot\n)\n\n\nfig.show(renderer=\"kaggle\")\n\nfig_3 = go.Figure(fig) # to show the same fig in the Results section\n\n\nprint_distribution_comments(\"FGM2\")\nprint_distribution_comments(\"FGM3\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### What is the goal ratio per each team category?"},{"metadata":{"scrolled":false,"trusted":true},"cell_type":"code","source":"label_colors = sns.color_palette(\"colorblind\").as_hex()[0:3]\n\nfig = make_subplots(rows=2, cols=1,\n shared_xaxes=True, vertical_spacing = 0.15)\n\ntwo_point_x = [] # default data, all X values\nthree_point_x = [] # data to use on button click, all X values\n\ntwo_point_text = [] # default text on the bar\nthree_point_text = [] # text on the bar on button click\n\nrow = 1 # row nr. for subplot\nfor df, won_df in zip(detailed_results_dfs, winning_dfs): # Make plots for both regular season and tournaments:\n\n i = 0 # counter for labels\n for label in order: # 'Ordinary', 'Cinderella', 'Top'\n \n plot_df = df[df.LABEL == label] # all games \n won_plot_df = won_df[won_df.LABEL == label] # games won\n \n \n # All games (visible) background layer:\n background_x = plot_df['FGA2'].mean()\n front_x = plot_df['FGM2'].mean()\n fig.add_trace(\n go.Bar(x=[background_x], # just one value for a bar\n y=[label],\n name=label,\n marker_color=label_colors[i],\n visible=True,\n opacity=0.5,\n orientation='h'), \n row=row, col=1)\n two_point_x.append([background_x])\n three_point_x.append([plot_df['FGA3'].mean()])\n \n two_point_text.append(\"\") # empty because there is no text in background bar\n three_point_text.append(\"\")\n \n # All games (visible) front layer:\n fig.add_trace(\n go.Bar(x=[front_x], # just one value for a bar\n y=[label],\n name=label,\n marker_color=label_colors[i],\n visible=True,\n orientation='h',\n \n text=(front_x/background_x), # calculate the ratio\n textposition='auto',\n texttemplate='%{text:.1%}'), # format output\n row=row, col=1)\n two_point_x.append([front_x])\n three_point_x.append([plot_df['FGM3'].mean()])\n \n two_point_text.append(front_x/background_x)\n three_point_text.append(plot_df['FGM3'].mean()/plot_df['FGA3'].mean())\n\n \n # Games won (not visible by default) background layer:\n background_x = won_plot_df['FGA2'].mean()\n front_x = won_plot_df['FGM2'].mean()\n fig.add_trace(\n go.Bar(x=[background_x], # just one value for a bar\n y=[label],\n name=label,\n marker_color=label_colors[i],\n visible=False,\n opacity=0.5,\n orientation='h'),\n row=row, col=1)\n two_point_x.append([background_x])\n three_point_x.append([won_plot_df['FGA3'].mean()]) # not visible\n \n two_point_text.append(\"\") # empty because there is no text in background bar\n three_point_text.append(\"\")\n \n # Games won (not visible by default) front layer:\n fig.add_trace(\n go.Bar(x=[front_x], # just one value for a bar\n y=[label],\n name=label,\n marker_color=label_colors[i],\n visible=False,\n orientation='h',\n \n text=(front_x/background_x), # calculate the ratio\n textposition='auto',\n texttemplate='%{text:.1%}'), # format output\n row=row, col=1)\n two_point_x.append([front_x])\n three_point_x.append([won_plot_df['FGM3'].mean()])\n \n two_point_text.append(front_x/background_x)\n three_point_text.append(won_plot_df['FGM3'].mean()/won_plot_df['FGA3'].mean())\n\n i+=1\n \n row+=1 # go to next subplot\n\nfig.update_layout(barmode='overlay') # the bars are plotted over one another\n\n# Default visibility:\nshow_all_games = [True, True, False, False, # 'Ordinary'\n True, True, False, False, # 'Cinderella'\n True, True, False, False, # 'Top'\n True, True, False, False, # 'Ordinary'\n True, True, False, False, # 'Cinderella'\n True, True, False, False] # 'Top'\n\n# Opposite visibility (reverse above list):\nshow_games_won = show_all_games[::-1]\n \nfig.update_layout(showlegend=False, # hide ledend\n width=plotly_width, height=550) # set size\n\n# Set axis font:\nfig.update_yaxes(tickfont=dict(size=14))\n\ndefault_title = \"Mean 2-point field goal ratio vs. team category,
2003-2019. Interactive graph.\"\nhidden_title = \"Mean 3-point field goal ratio vs. team category,
2003-2019. Interactive graph.\"\n\ndefault_xtitle = dict(x=0.5, y=-0.15, xref=\"paper\", yref=\"paper\",\n text=\"Mean 2-point goals per game (scored / attempted)\",\n showarrow=False, font=dict(size=14))\nhidden_xtitle = dict(x=0.5, y=-0.15, xref=\"paper\", yref=\"paper\",\n text=\"Mean 3-point goals per game (scored / attempted)\",\n showarrow=False, font=dict(size=14))\n\nupper_subplot_title = dict(x=0.5, y=1.075, xref=\"paper\", yref=\"paper\",\n text=\"Regular season\", showarrow=False, font=dict(size=16))\nlower_subplot_title = dict(x=0.5, y=0.475, xref=\"paper\", yref=\"paper\",\n text=\"Tournaments\", showarrow=False, font=dict(size=16))\n\n# Add subplot titles:\nfig.add_annotation(upper_subplot_title)\nfig.add_annotation(lower_subplot_title)\n\n# Add annotations:\nfig.add_annotation(default_xtitle)\n\ndefault_annotations = [default_xtitle, upper_subplot_title, lower_subplot_title]\nhidden_annotations = [hidden_xtitle, upper_subplot_title, lower_subplot_title]\n\n# Add buttons:\nfig.update_layout(\n updatemenus=[\n dict( # these buttons will change data\n type=\"buttons\",\n direction=\"right\",\n active=0,\n x=0.45,\n y=1.2,\n buttons=list([\n dict(label=\"2-point goals\",\n method=\"update\",args=[{\"x\": two_point_x, \"text\": two_point_text},\n {\"title\": default_title,\n \"annotations\": default_annotations},\n {\"visible\": show_all_games}]), \n dict(label=\"3-point goals\",\n method=\"update\",args=[{\"x\": three_point_x, \"text\": three_point_text},\n {\"title\": hidden_title,\n \"annotations\": hidden_annotations},\n {\"visible\": show_all_games}])\n ]),\n ),\n dict( # these buttons will change visibility of \"games won\"\n buttons=list([\n dict(label=\"All games\",\n method=\"restyle\",args=[{\"visible\": show_all_games}]), \n dict(label=\"Games won\",\n method=\"restyle\",args=[{\"visible\": show_games_won}])\n ]),\n direction=\"down\",\n showactive=True,\n x=0.8,\n y=1.2,\n )\n ])\n\n# Plot title:\nfig.update_layout(\n title={\n 'text': default_title,\n 'y':0.95,\n 'x':0.5,\n 'xanchor': 'center',\n 'yanchor': 'top'},\n font=dict(\n family='sans-serif',\n color=\"#000\"),\n margin=dict(t=150) # margin between title and plot\n)\n\n\nfig.show(renderer=\"kaggle\")\n\nfig_5 = go.Figure(fig) # to show the same fig in the Results section\n\n\nprint_distribution_comments(\"FGA2\")\nprint_distribution_comments(\"FGA3\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### What is the assist / turnover ratio per each team category?"},{"metadata":{"trusted":true},"cell_type":"code","source":"label_colors = sns.color_palette(\"colorblind\").as_hex()[0:3]\n\nfig = make_subplots(rows=2, cols=1, subplot_titles=(\"Regular season\", \"Tournaments\"),\n shared_xaxes=True, vertical_spacing = 0.15)\n\nrow = 1 # row nr. for subplot\nfor df, won_df in zip(detailed_results_dfs, winning_dfs): # Make plots for both regular season and tournaments:\n\n i = 0 # counter for labels\n for label in order: # 'Ordinary', 'Cinderella', 'Top'\n \n plot_df = df[df.LABEL == label] # all games \n won_plot_df = won_df[won_df.LABEL == label] # games won\n \n \n # All games - visible:\n background_x = plot_df['Ast'].mean()\n front_x = plot_df['TO'].mean()\n \n def plot_bar(background_x, front_x, visible):\n # Upper bar:\n fig.add_trace(\n go.Bar(x=[background_x], # just one value for a bar\n y=[label],\n name=label,\n marker_color=label_colors[i],\n visible=visible,\n opacity=1,\n orientation='h', width=0.35, offset=-0.05,\n\n text=(background_x/front_x), # calculate the ratio\n textposition='outside',\n texttemplate='%{text:.2f}'), # format output \n row=row, col=1)\n # Lower bar:\n fig.add_trace(\n go.Bar(x=[front_x], # just one value for a bar\n y=[label],\n name=label,\n marker_color=label_colors[i],\n visible=visible,\n opacity=0.35,\n orientation='h', width=0.35, offset=-0.40),\n row=row, col=1)\n\n plot_bar(background_x, front_x, True)\n \n\n # Games won - not visible by default:\n background_x = won_plot_df['Ast'].mean()\n front_x = won_plot_df['TO'].mean()\n \n plot_bar(background_x, front_x, False)\n\n i+=1\n \n row+=1 # go to next subplot\n \n \n# Default visibility:\nshow_all_games = [True, True, False, False, # 'Ordinary'\n True, True, False, False, # 'Cinderella'\n True, True, False, False, # 'Top'\n True, True, False, False, # 'Ordinary'\n True, True, False, False, # 'Cinderella'\n True, True, False, False] # 'Top'\n\n# Opposite visibility (reverse above list):\nshow_games_won = show_all_games[::-1]\n \nfig.update_layout(showlegend=False, # hide ledend\n width=plotly_width, height=650) # set size\n\n# Set axis font:\nfig.update_yaxes(tickfont=dict(size=14))\n\n# Add titles:\nfig.update_xaxes(title_text='Mean assists / turnovers', row=2, col=1)\n\n# Add buttons:\nfig.update_layout(\n updatemenus=[\n dict(\n type=\"buttons\",\n direction=\"right\",\n active=0,\n x=0.6,\n y=1.2,\n buttons=list([\n dict(label=\"All games\",\n method=\"restyle\",args=[{\"visible\": show_all_games}]), \n dict(label=\"Games won\",\n method=\"restyle\",args=[{\"visible\": show_games_won}])\n ]),\n )\n ])\n\n# Plot title:\nfig.update_layout(\n title={\n 'text': \"Mean Assist to Turnover Ratio vs. team category,
2003-2019. Interactive graph.\",\n 'y':0.95,\n 'x':0.5,\n 'xanchor': 'center',\n 'yanchor': 'top'},\n font=dict(\n family='sans-serif',\n color=\"#000\"),\n margin=dict(t=150) # margin between title and plot\n)\n\n\nfig.show(renderer=\"kaggle\")\nfig_6 = go.Figure(fig) # to show the same fig in the Results section","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Are Cinderellas a good defenders?"},{"metadata":{"trusted":true},"cell_type":"code","source":"palette = [sns.color_palette(\"cubehelix\", 10).as_hex()[6], sns.color_palette(\"cubehelix\", 10).as_hex()[1], 'gold']\n\nlabel_colors_a = [palette[0], palette[0], palette[0]]\nlabel_colors_b = [palette[1], palette[1], palette[1]]\nlabel_colors_c = [palette[2], palette[2], palette[2]]\n\nfig = make_subplots(rows=2, cols=1, subplot_titles=(\"Regular season\", \"Tournaments\"),\n shared_xaxes=True, vertical_spacing = 0.15)\n\nrow = 1 # row nr. for subplot\nfor df, won_df in zip(detailed_results_dfs, winning_dfs): # Make plots for both regular season and tournaments:\n\n i = 0 # counter for labels\n for label in order: # 'Ordinary', 'Cinderella', 'Top'\n \n plot_df = df[df.LABEL == label] # all games \n won_plot_df = won_df[won_df.LABEL == label] # games won\n \n \n # All games - visible:\n colors = [label_colors_a, label_colors_b, label_colors_c]\n scores = ['DR
', 'STL
', 'BLK
'] # text to show\n \n # Move common code to a function to reuse multiple times:\n def plot_bar(df, scores, colors, visible):\n \n x_list = [df['DR'].mean(), df['Stl'].mean(), df['Blk'].mean()]\n \n for x, score, colors in zip(x_list, scores, colors):\n fig.add_trace(\n go.Bar(x=[x], # just one number value for a bar\n y=[label],\n name=label,\n marker_color=colors[i],\n visible=visible,\n opacity=1,\n orientation='h',\n\n text=(x), \n textposition='inside',\n texttemplate=score + '%{text:.2f}'),\n row=row, col=1)\n \n\n plot_bar(plot_df, scores, colors, True)\n \n\n # Games won - not visible by default:\n\n plot_bar(won_plot_df, scores, colors, False)\n\n i+=1\n \n row+=1 # go to next subplot\n \nfig.update_layout(barmode='stack')\n \n# Controlling text fontsize with uniformtext\nfig.update_layout(uniformtext_minsize=12, uniformtext_mode='show')\n\n\n \nfig.update_layout(showlegend=False, # hide ledend\n width=plotly_width, height=550) # set size\n\n# Set axis font:\nfig.update_yaxes(tickfont=dict(size=14))\n\n# Add titles:\nfig.update_xaxes(title_text='DR - defensive rebounds | STL - steals | BLK - blocks', row=2, col=1)\n\n### BUTTONS ###\n\n# Default visibility:\n\none_label_visibility = [[True]*3, [False]*3] # all bars for one label\none_subplot_visibility = one_label_visibility*3 # all bars for one subplot\nshow_all_games = sum(one_label_visibility*2, []) # all bars for two subplots converted to one list\n\n# Opposite visibility (reverse above list):\nshow_games_won = show_all_games[::-1]\n\n# Add buttons:\nfig.update_layout(\n updatemenus=[\n dict(\n type=\"buttons\",\n direction=\"right\",\n active=0,\n x=0.6,\n y=1.2,\n buttons=list([\n dict(label=\"All games\",\n method=\"restyle\",args=[{\"visible\": show_all_games}]), \n dict(label=\"Games won\",\n method=\"restyle\",args=[{\"visible\": show_games_won}])\n ]),\n )\n ])\n\n### END BUTTONS ###\n\n# Plot title:\nfig.update_layout(\n title={\n 'text': \"Mean defence statistics per game vs. team category,
2003-2019. Interactive graph.\",\n 'y':0.95,\n 'x':0.5,\n 'xanchor': 'center',\n 'yanchor': 'top'},\n font=dict(\n family='sans-serif',\n color=\"#000\"),\n margin=dict(t=150) # margin between title and plot\n)\n\n\nfig.show(renderer=\"kaggle\")\nfig_7 = go.Figure(fig) # to show the same fig in the Results section\n\n\nprint_distribution_comments(\"DR\")\nprint(\"\\n***\")\nprint_distribution_comments(\"Stl\")\nprint(\"\\n***\")\nprint_distribution_comments(\"Blk\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Are Cinderella teams able to defend without fouling?"},{"metadata":{"trusted":true},"cell_type":"code","source":"label_colors = sns.color_palette(\"colorblind\").as_hex()[0:3]\n\nfig = make_subplots(rows=2, cols=1, subplot_titles=(\"Regular season\", \"Tournaments\"),\n shared_xaxes=True, vertical_spacing = 0.15)\n\nrow = 1 # row nr. for subplot\nfor df, won_df in zip(detailed_results_dfs, winning_dfs): # Make plots for both regular season and tournaments:\n\n i = 0 # counter for labels\n for label in order: # 'Ordinary', 'Cinderella', 'Top'\n \n plot_df = df[df.LABEL == label] # all games \n won_plot_df = won_df[won_df.LABEL == label] # games won\n \n # Move common code to a function to reuse multiple times:\n def plot_bar(df, visible):\n \n background_x = df['PF'].mean()\n front_x = df['Blk'].mean()\n \n # Background layer:\n fig.add_trace(\n go.Bar(x=[background_x], # just one value for a bar\n y=[label],\n name=label,\n marker_color=label_colors[i],\n visible=visible,\n opacity=0.5,\n orientation='h'), \n row=row, col=1)\n # Front layer:\n fig.add_trace(\n go.Bar(x=[front_x], # just one value for a bar\n y=[label],\n name=label,\n marker_color=label_colors[i],\n visible=visible,\n orientation='h',\n\n text=(front_x/background_x), # calculate the ratio\n textposition='auto',\n texttemplate='%{text:.1%}'), # format output\n row=row, col=1)\n \n \n # All games:\n plot_bar(plot_df, True)\n \n # Games won:\n plot_bar(won_plot_df, False)\n\n i+=1\n \n row+=1 # go to next subplot\n\nfig.update_layout(barmode='overlay') # the bars are plotted over one another\n \n# Default visibility:\nshow_all_games = [True, True, False, False, # 'Ordinary'\n True, True, False, False, # 'Cinderella'\n True, True, False, False, # 'Top'\n True, True, False, False, # 'Ordinary'\n True, True, False, False, # 'Cinderella'\n True, True, False, False] # 'Top'\n\n# Opposite visibility (reverse above list):\nshow_games_won = show_all_games[::-1]\n \nfig.update_layout(showlegend=False, # hide ledend\n width=plotly_width, height=550) # set size\n\n# Set axis font:\nfig.update_yaxes(tickfont=dict(size=14))\n\n# Add titles:\nfig.update_xaxes(title_text='Mean blocks / personal fouls', row=2, col=1)\n\n# Add buttons:\nfig.update_layout(\n updatemenus=[\n dict(\n type=\"buttons\",\n direction=\"right\",\n active=0,\n x=0.6,\n y=1.2,\n buttons=list([\n dict(label=\"All games\",\n method=\"restyle\",args=[{\"visible\": show_all_games}]), \n dict(label=\"Games won\",\n method=\"restyle\",args=[{\"visible\": show_games_won}])\n ]),\n )\n ])\n\n# Plot title:\nfig.update_layout(\n title={\n 'text': \"Mean blocks per fouls vs. team category,
2003-2019. Interactive graph.\",\n 'y':0.95,\n 'x':0.5,\n 'xanchor': 'center',\n 'yanchor': 'top'},\n font=dict(\n family='sans-serif',\n color=\"#000\"),\n margin=dict(t=150) # margin between title and plot\n)\n\n\nfig.show(renderer=\"kaggle\")\nfig_8 = go.Figure(fig) # to show the same fig in the Results section\n\n\nprint_distribution_comments(\"PF\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### How tough are Cinderellas around the rim?"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = make_subplots(rows=2, cols=1, subplot_titles=(\"Regular season\", \"Tournaments\"),\n shared_xaxes=True, vertical_spacing = 0.15)\n\nrow = 1 # row nr. for subplot\nfor df, won_df in zip(detailed_results_dfs, winning_dfs): # Make plots for both regular season and tournaments:\n\n i = 0\n for label in order: # 'Ordinary', 'Cinderella', 'Top'\n \n plot_df = df[df.LABEL == label] # all games \n won_plot_df = won_df[won_df.LABEL == label] # games won\n \n # All games (visible):\n fig.add_trace(\n go.Box(x=plot_df['Rebound Margin'],\n name=label,\n marker_color=sns.color_palette(\"colorblind\").as_hex()[i],\n boxmean=True, # represent mean\n boxpoints='suspectedoutliers',\n visible=True), # only suspected outliers\n row=row, col=1)\n \n # Games won (not visible by default):\n fig.add_trace(\n go.Box(x=won_plot_df['Rebound Margin'],\n name=label,\n marker_color=sns.color_palette(\"colorblind\").as_hex()[i],\n boxmean=True, # represent mean\n boxpoints='suspectedoutliers', # only suspected outliers\n visible=False), \n row=row, col=1)\n \n i+=1\n \n row+=1 # go to next subplot\n\n# Add vertical line to represent zero Rebound Margin:\nfig.update_layout(\n shapes=[\n dict(type=\"line\", xref=\"x1\", yref=\"y1\", # col 1, row 1\n x0=0, y0=-1, x1=0, opacity=0.5,\n line=dict(dash='dash', color='grey')),\n dict(type=\"line\", xref=\"x1\", yref=\"y2\", # col 1, row 2\n x0=0, y0=-1, x1=0, opacity=0.5,\n line=dict(dash='dash', color='grey'))])\n \nfig.update_layout(showlegend=False, # hide ledend\n width=plotly_width, height=750) # set size\n\n# Set axis font:\nfig.update_yaxes(tickfont=dict(size=14))\n\n# Add titles:\nfig.update_xaxes(title_text='Rebound Margin per game', row=2, col=1)\n\n# Default visibility:\nshow_all_games = [True, False, True, False, True, False, # row 1: OO CC TT ('Ordinary', 'Cinderella', 'Top')\n True, False, True, False, True, False] # row 2: OO CC TT ('Ordinary', 'Cinderella', 'Top')\n\n# Opposite visibility (reverse above list):\nshow_games_won = show_all_games[::-1]\n\n# Add buttons:\nfig.update_layout(\n updatemenus=[\n dict(\n type=\"buttons\",\n direction=\"right\",\n active=0,\n x=0.6,\n y=1.2,\n buttons=list([\n dict(label=\"All games\",\n method=\"restyle\",args=[{\"visible\": show_all_games}]), \n dict(label=\"Games won\",\n method=\"restyle\",args=[{\"visible\": show_games_won}])\n ]),\n )\n ])\n\n\n# Plot title:\nfig.update_layout(\n title={\n 'text': \"Rebound Margin (distribution) vs. team category,
2003-2019. Interactive graph.\",\n 'y':0.95,\n 'x':0.5,\n 'xanchor': 'center',\n 'yanchor': 'top'},\n font=dict(\n family='sans-serif',\n color=\"#000\"),\n margin=dict(t=180) # margin between title and plot\n)\n\n\nfig.show(renderer=\"kaggle\")\nfig_10 = go.Figure(fig) # to show the same fig in the Results section\n\n\nprint_distribution_comments('Rebound Margin')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"---\n## 2.4. Play-by-play, 2015-2020\n\n**Data Section 5 files: MEvents2015.csv, MEvents2016.csv, MEvent2017.csv, MEvents2018.csv, MEvents2019.csv, MEvents2020.csv** - each MEvents file lists the play-by-play event logs for more than 99.5% of games from that season. Each event is assigned to either a team or a single one of the team's players.\n\nEventTeamID - this is the ID of the team that the event is logged for, which will either be the WTeamID or the LTeamID [[1]](#References)."},{"metadata":{"trusted":true},"cell_type":"code","source":"events_dir = '/kaggle/input/march-madness-analytics-2020/MPlayByPlay_Stage2/'\n\nif sys.executable != '/opt/conda/bin/python':\n # remove the forward slash if running this notebook locally:\n events_dir = events_dir[1:]\n\ndef load_events_file(df, name):\n '''Load in the file and show basic info'''\n print(\"File: {}\".format(name))\n df = pd.read_csv(events_dir + name + '.csv')\n print(\"Num rows: {}\".format(len(df)))\n print(\"NaN values: {}\".format(df.isna().sum().sum()))\n print(\"Duplicated rows: {}\".format(df.duplicated().sum()))\n print(list(df))\n print(\"\\n\")\n return(df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Load in the files:"},{"metadata":{"trusted":true},"cell_type":"code","source":"MEvents2015 = None\nMEvents2015 = load_events_file(MEvents2015, 'MEvents2015')\n\nMEvents2016 = None\nMEvents2016 = load_events_file(MEvents2016, 'MEvents2016')\n\nMEvents2017 = None\nMEvents2017 = load_events_file(MEvents2017, 'MEvents2017')\n\nMEvents2018 = None\nMEvents2018 = load_events_file(MEvents2018, 'MEvents2018')\n\nMEvents2019 = None\nMEvents2019 = load_events_file(MEvents2019, 'MEvents2019')\n\nMEvents2020 = None\nMEvents2020 = load_events_file(MEvents2020, 'MEvents2020')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"MEvents2015","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Just a test:\npd.concat([MEvents2020.head(3), MEvents2020.tail(2)])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Make one common MEvents dataframe via concatenating 6 files together**"},{"metadata":{"trusted":true},"cell_type":"code","source":"MEvents = pd.concat([MEvents2015, MEvents2016, MEvents2017, MEvents2018, MEvents2019, MEvents2020],\n sort=False, ignore_index=True)\n\nprint(\"Play-by-play event logs (all 15835846 logged events):\")\nMEvents","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Make a separate dataframe for labeled NCAA® tournament events**"},{"metadata":{"scrolled":true,"trusted":true},"cell_type":"code","source":"len(labeled_MNCAATourneyCompactResults[labeled_MNCAATourneyCompactResults.Season.isin([2015,2016,2017,2018,2019])])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\ncols = ['Season', 'DayNum', 'WTeamID', 'LTeamID']\nlabeled_tourney_MEvents = MEvents.join(labeled_MNCAATourneyCompactResults.set_index(cols), on=cols, how='inner')\n\nprint(len(labeled_tourney_MEvents.groupby(cols).sum())) # should be 335\n\nlabeled_tourney_MEvents","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"min(labeled_tourney_MEvents.DayNum), max(labeled_tourney_MEvents.DayNum) # just a test","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Are missing values in X, Y encoded as zeros?"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1,2, figsize = (14, 6))\n\nax[0].hist(MEvents['X'], bins=20)\n\nax[1].hist(MEvents[MEvents['X'] != 0]['X'], bins=20)\n\nax[0].set_title(\"Including zeros\")\nax[1].set_title(\"Without zeros\")\n\nplt.suptitle(\"Distribution of X coordinate values in MEvents dataframe.\", y=1.05)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Get rid of zero coordinates**\n\nSelect rows where \"X\" is not zero:"},{"metadata":{"trusted":true},"cell_type":"code","source":"court_MEvents = MEvents[MEvents.X != 0]\n\ncourt_labeled_tourney_MEvents = labeled_tourney_MEvents[labeled_tourney_MEvents['X'] != 0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"What events are available with coordinates?"},{"metadata":{"trusted":true},"cell_type":"code","source":"# What events are available with coordinates?\ncourt_MEvents.EventType.value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Which season data has events with coordinates?"},{"metadata":{"trusted":true},"cell_type":"code","source":"court_MEvents.Season.value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### What is the player location for most three-point shots?"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Court outline (image source: [25])\nline_img = plt.imread(\"https://raw.githubusercontent.com/evanca/data-analysis_kaggle_march-madness-analytics-2020/master/img/Vve3bT9.png\") ","execution_count":null,"outputs":[]},{"metadata":{"scrolled":false,"trusted":true},"cell_type":"code","source":"df = court_MEvents[court_MEvents.EventType == 'made3']\n\nfig, ax = plt.subplots(figsize=(14,7.5))\n\nsns.kdeplot(df['X'], df['Y'], shade=True, cmap='Reds',\n n_levels=25, alpha=1).set(xlim=(0, 100), ylim=(0, 100))\n\nax.imshow(line_img, extent=[0, 100, 0, 100], aspect='auto', alpha=0.15, zorder=10)\n\n# Remove coordinates:\nax.get_xaxis().set_visible(False)\nax.get_yaxis().set_visible(False)\n\nplt.title('Three-point goal heatmap,\\n2019-2020.\\n')\nplt.show()\n\n\nprint(\"\\n Area nr. vs. three-point goals:\")\ndf = pd.DataFrame(court_MEvents[court_MEvents.EventType == 'made3']['Area'].value_counts())\ndf['Share'] = df['Area'] / sum(df['Area'])\ndf.columns=['Sum', 'Share']\ndf = df.style.format({'Share': \"{:.2%}\"})\ndisplay(df)\n\nprint(\"\\n9 = outside right\\n10 = outside center\\n11 = outside left\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### What is the player location for most turnovers?"},{"metadata":{"trusted":true},"cell_type":"code","source":"df = court_MEvents[(court_MEvents.EventType == 'turnover') &\n (court_MEvents.Area.isin([8,9,10,11,12]))]\n\nfig, ax = plt.subplots(figsize=(14,7.5))\n\nsns.kdeplot(df['X'], df['Y'], shade=True, cmap='Blues',\n n_levels=25, alpha=1).set(xlim=(0, 100), ylim=(0, 100))\n\nax.imshow(line_img, extent=[0, 100, 0, 100], aspect='auto', alpha=0.15, zorder=10)\n\n# Remove coordinates:\nax.get_xaxis().set_visible(False)\nax.get_yaxis().set_visible(False)\n\nplt.title('Turnover heatmap beyond the three-point line,\\n2019-2020.\\n')\nplt.show()\n\nprint(\"\\n Area nr. vs. turnovers:\")\ndf = pd.DataFrame(court_MEvents[(court_MEvents.EventType == 'turnover')&\n (court_MEvents.Area.isin([8,9,10,11,12]))]['Area'].value_counts())\ndf['Share'] = df['Area'] / sum(df['Area'])\ndf.columns=['Sum', 'Share']\ndf = df.style.format({'Share': \"{:.2%}\"})\ndisplay(df)\n\nprint(\"\\n9 = outside right\\n10 = outside center\\n11 = outside left\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Make a final image to show in the [Results](#II.-Results) section:"},{"metadata":{"scrolled":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize = (14, 15))\n\n\n### Plot 1 ###\n\nax1 = fig.add_axes([0, 0.5, 0.8, 0.4]) # [left, bottom, width, height]\n\ndf = court_MEvents[court_MEvents.EventType == 'made3']\n\nsns.kdeplot(df['X'], df['Y'], shade=True, cmap='Reds',\n n_levels=25, alpha=1, ax=ax1).set(xlim=(0, 100), ylim=(0, 100))\n\nax1.imshow(line_img, extent=[0, 100, 0, 100], aspect='auto', alpha=0.15, zorder=10)\n\n# Remove coordinates:\nax1.get_xaxis().set_visible(False)\nax1.get_yaxis().set_visible(False)\n\nax1.set_title('Figure 5. Three-point goal heatmap,\\n2018-19 and 2019-20.\\n')\n\n\n\n### Plot 2 ###\n\nax2 = fig.add_axes([0, 0, 0.8, 0.4]) # [left, bottom, width, height]\n\ndf = court_MEvents[(court_MEvents.EventType == 'turnover') &\n (court_MEvents.Area.isin([8,9,10,11,12]))]\n\nsns.kdeplot(df['X'], df['Y'], shade=True, cmap='Blues',\n n_levels=25, alpha=1, ax=ax2).set(xlim=(0, 100), ylim=(0, 100))\n\nax2.imshow(line_img, extent=[0, 100, 0, 100], aspect='auto', alpha=0.15, zorder=10)\n\n# Remove coordinates:\nax2.get_xaxis().set_visible(False)\nax2.get_yaxis().set_visible(False)\n\nax2.set_title('Figure 6. Turnover heatmap beyond the three-point line,\\n2018-19 and 2019-20.\\n')\n\nsave_plot()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"---\n### When are the most field goals scored?\n\n**Create 5 minute bins for ElapsedSeconds**"},{"metadata":{"trusted":true},"cell_type":"code","source":"binned_tourney_MEvents = labeled_tourney_MEvents\n\n# Create interval for bins\ninterval_range = pd.interval_range(start=0, freq=300, end=binned_tourney_MEvents['ElapsedSeconds'].max())\n\n# Create a bin column\nbinned_tourney_MEvents['bin'] = pd.cut(labeled_tourney_MEvents['ElapsedSeconds'], interval_range)\n\nassert len(binned_tourney_MEvents) == len(labeled_tourney_MEvents)\n\nbinned_tourney_MEvents.sample(3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# The same but without Nulls:\nlen(binned_tourney_MEvents[binned_tourney_MEvents['bin'].notna()])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Make a column with minute values (instead of seconds):"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Clean the data: remove Null bins\nbinned_tourney_MEvents = binned_tourney_MEvents[binned_tourney_MEvents['bin'].notna()]\n\n# Make a column with minute values (instead of seconds)\n\nbinned_tourney_MEvents['bin'] = binned_tourney_MEvents['bin'].astype(str)\nbinned_tourney_MEvents = pd.concat([binned_tourney_MEvents, binned_tourney_MEvents['bin'].str.split(', ', expand=True)], axis=1)\n\nbinned_tourney_MEvents.sample(3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Format \"bin\" column:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Format \"bin\" column to \"nr - nr\" output:\n\nbinned_tourney_MEvents[0] = (binned_tourney_MEvents[0].str.extract(r'(\\d+)').astype(int)/60).astype(int) # extract numbers and convert to minutes\nbinned_tourney_MEvents[1] = (binned_tourney_MEvents[1].str.extract(r'(\\d+)').astype(int)/60).astype(int)\n\nbinned_tourney_MEvents = binned_tourney_MEvents.sort_values(0) # sort by bins\n\nbinned_tourney_MEvents['bin'] = binned_tourney_MEvents[0].astype(str) + \" - \" + binned_tourney_MEvents[1].astype(str)\n\nbinned_tourney_MEvents.sample(3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Calculate total field goals vs. elapsed time**\n\n- Filter out 2-pointers and 3-pointers\n- Group data by time bin\n- Count events in each bin"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Count EventID cells for each column:\nfield_goals = binned_tourney_MEvents[(binned_tourney_MEvents['EventType'] == 'made2') |\n (binned_tourney_MEvents['EventType'] == 'made3')][['Season', 'EventID',\n 'bin']].groupby(['Season', 'bin'],\n as_index=False,\n sort=False).count() \n\nfield_goals.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create figure\nfig = go.Figure()\n\n# Add traces, one for each slider step\nall_seasons = [2015, 2016, 2017, 2018, 2019]\n\n# First trace - step [0] with all data:\ndf = field_goals.groupby('bin', as_index=False, sort=False).sum()\n\ny=df['EventID']\n\n# Different color for the biggest column:\ncolor=np.array([sns.color_palette(\"cubehelix\", 10).as_hex()[6]]*y.shape[0])\ncolor[y < max(y)]=sns.color_palette(\"cubehelix\", 10).as_hex()[5]\n\nfig.add_trace(\n go.Bar(\n visible=False,\n x=df['bin'],\n y=y,\n marker_color=color.tolist(),\n \n text=(y),\n textposition='outside')) \n\n\n# Next 5 steps by season:\nfor season in all_seasons: \n \n df = field_goals[field_goals.Season == season]\n \n y=df['EventID']\n \n # Different color for the biggest column:\n color=np.array([sns.color_palette(\"cubehelix\", 10).as_hex()[6]]*y.shape[0])\n color[y < max(y)]=sns.color_palette(\"cubehelix\", 10).as_hex()[5]\n\n \n fig.add_trace(\n go.Bar(\n visible=False,\n x=df['bin'],\n y=y,\n marker_color=color.tolist(),\n \n text=(y),\n textposition='outside')) \n\n# Make 0th trace visible\nfig.data[0].visible = True\n\n\n### ADD SLIDER ###\nsteps = []\nstep_labels = ['ALL
(2015-2019)', '2015', '2016', '2017', '2018', '2019']\nfor i in range(len(fig.data)):\n step = dict(\n label=step_labels[i],\n method=\"restyle\",\n args=[\"visible\", [False] * len(fig.data)],\n )\n step[\"args\"][1][i] = True # Toggle i'th trace to \"visible\"\n steps.append(step)\n\nsliders = [dict(\n active=0,\n currentvalue={\"prefix\": \"Season: \"},\n pad={\"t\": 50},\n steps=steps\n)]\n\nfig.update_layout(\n sliders=sliders\n)\n### END SLIDER ###\n\nfig.update_layout(showlegend=False, # hide ledend\n width=plotly_width, height=750) # set size\n\n# Set axis font:\nfig.update_yaxes(tickfont=dict(size=14))\n\n# Add titles:\nfig.update_xaxes(title_text='Elapsed time, minutes')\nfig.update_yaxes(title_text='Total goals')\n\n# Plot title:\nfig.update_layout(\n title={\n 'text': \"Field goals scored vs. elapsed time,
2015-2019 NCAA® tournaments. Interactive graph.\",\n 'y':0.95,\n 'x':0.5,\n 'xanchor': 'center',\n 'yanchor': 'top'},\n font=dict(\n family='sans-serif',\n color=\"#000\")\n)\n\n\nfig.show(renderer=\"kaggle\")\n\nfig_1 = go.Figure(fig) # to show the same fig in the Results section","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"---\n\n### How does the goal accuracy change with the distance?"},{"metadata":{"scrolled":false,"trusted":true},"cell_type":"code","source":"from matplotlib.lines import Line2D\n\ncolors = [sns.color_palette(\"cubehelix\", 10)[6], sns.color_palette(\"cubehelix\", 10)[1], 'gold']\n\nfig, ax = plt.subplots(figsize=(14,7.5))\n\n# Show background image:\nax.imshow(line_img, extent=[0, 100, 0, 100], aspect='auto', alpha=0.5)\n\n\nsns.scatterplot(x='X', y='Y', data=court_labeled_tourney_MEvents[court_labeled_tourney_MEvents.EventType == 'miss2'],\n alpha=0.35,\n edgecolor=None,\n color=colors[1]).set(xlim=(0, 100), ylim=(0, 100))\n\nsns.scatterplot(x='X', y='Y', data=court_labeled_tourney_MEvents[court_labeled_tourney_MEvents.EventType == 'made2'],\n alpha=0.35,\n edgecolor=None,\n color=colors[0]).set(xlim=(0, 100), ylim=(0, 100))\n\nsns.scatterplot(x='X', y='Y', data=court_labeled_tourney_MEvents[court_labeled_tourney_MEvents.EventType == 'miss3'],\n alpha=0.35,\n edgecolor=None,\n color=colors[1]).set(xlim=(0, 100), ylim=(0, 100))\n\nsns.scatterplot(x='X', y='Y', data=court_labeled_tourney_MEvents[court_labeled_tourney_MEvents.EventType == 'made3'],\n alpha=0.35,\n edgecolor=None,\n color=colors[0]).set(xlim=(0, 100), ylim=(0, 100))\n\n\nax = plt.gca()\n\nax.legend(handles=[(Line2D([0],[0], marker='o', markerfacecolor=colors[0], \n linestyle='none', markersize=10, markeredgecolor='none')),\n (Line2D([0],[0], marker='o', markerfacecolor=colors[1],\n linestyle='none', markersize=10, markeredgecolor='none'))],\n labels=[\"goal made\", \"goal missed\"], loc=\"upper center\")\n\n\n# Remove coordinate values:\nax = plt.gca()\nax.get_xaxis().set_visible(False)\nax.get_yaxis().set_visible(False)\n\nplt.title(\"Figure 4. Field goal accuracy by player location,\\n2015-2019 NCAA® tournaments.\\n\")\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"---\n**Calculate goal distance in m based on X, Y coordinates**\n\nThe court has 15.2 m (50 ft) width and 28.7 m (94 feet) length, equivalent to 100X * 100Y. From this we can calculate one \"square\" size: 0.152m width x 0.287m length.\n\n2 points are awarded to players who successfully shoot the ball through the hoop from anywhere inside the three-point line. This can be done by shooting a jump shot, laying the ball into the rim, or slamming the ball through the hoop. 3 points are warded to players who successfully shoot the ball through the hoop from behind the three-point line [[26]](#References).\n\nNote that we will assume here that all 3 point goals were made from the same side of the court where the basket is. While this will be true for most cases, few insignificant errors are possible (three-pointer from the other side of the court).\n\nOur pseudocode:\n\n1. Filter out all field goal rows\n2. If X > 50, then our basket is on the right, else our basket is on the left side of the court\n3. Calculate distance from player to basket\n4. Convert coordinate distance to meters"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Filter out all field goal rows for NCAA® tournaments:\n\ngoals = court_labeled_tourney_MEvents[court_labeled_tourney_MEvents['EventType'].isin(['made2', 'miss2', 'made3', 'miss3'])]\ngoals","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"What are the coordinates of each basket?"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Right basket:\ngoals[(goals['Area'] == 1) & (goals['X'] > 50)][[\"X\", \"Y\"]].mean().round()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From this result, we can tell that **coordinates for the left basket are: X=7, Y=50**.\n\nNext, **convert our coordinates to meters** (as if our court was divided into 1x1m grid):"},{"metadata":{"trusted":true},"cell_type":"code","source":"goals['XMeters'] = goals['X']*0.287\ngoals['YMeters'] = goals['Y']*0.152\n\ngoals.sample(3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Basket coordinates in meters:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Basket coordinates in meters:\n\nright_basket_m = (93*0.287, 50*0.152)\nleft_basket_m = (7*0.287, 50*0.152)\n\nprint(right_basket_m, left_basket_m)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Calculate the distance**"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\nimport math\ndef calculate_distance(x1,y1, x2,y2): \n '''Calculate distance between two points'''\n dist = math.sqrt((x2 - x1)**2 + (y2 - y1)**2) \n return dist \n\ngoals['ShotDistanceMeters'] = None\n\nlen_goals = len(goals)\n\ni = 1\nfor index, row in goals.iterrows():\n if goals['X'][index] > 50: # right basket\n goals['ShotDistanceMeters'][index] = calculate_distance(goals['XMeters'][index],\n goals['YMeters'][index],\n right_basket_m[0],\n right_basket_m[1])\n elif goals['X'][index] < 50: # left basket\n goals['ShotDistanceMeters'][index] = calculate_distance(goals['XMeters'][index],\n goals['YMeters'][index],\n left_basket_m[0],\n left_basket_m[1])\n i+=1\n print(\"Updating row nr. {} / of {}\".format(i, len_goals)\n + \" \"*100, end=\"\\r\", flush=True) # erase output and print on the same line\n\nprint(\" \"*100, end=\"\\r\", flush=True) # erase final output\ngoals.sample(3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"goals['ShotDistanceMeters'].isna().sum() # just a test","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Split distance to 0.5 m bins:"},{"metadata":{"trusted":true},"cell_type":"code","source":"min(goals['ShotDistanceMeters']), max(goals['ShotDistanceMeters'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from operator import attrgetter\n\n# Create interval for bins\ninterval_range = pd.interval_range(start=-0.5, freq=0.5, end=12)\n\n# Create a bin column\ngoals['DistanceBin'] = pd.cut(goals['ShotDistanceMeters'], interval_range)\n\n# Note the right number of interval (a square bracket is inclusive) \ngoals['DistRightBound'] = goals['DistanceBin'].map(attrgetter('right'))\n\ngoals.sample(3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Calculate shot accuracy per distance in meters**"},{"metadata":{"trusted":true},"cell_type":"code","source":"df = goals.groupby(['DistRightBound', 'LABEL', 'EventType'], as_index=False)['EventID'].count()\ndf","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"One-hot encode \"made\" or \"miss\":"},{"metadata":{"trusted":true},"cell_type":"code","source":"dummies = pd.get_dummies(df['EventType']) # what type of event\n\nfor col in dummies:\n df[col] = dummies[col]*df['EventID'] # how many of such events\n\ndf[~df.EventID.isna()]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Calculate field-goal percentage (goal accuracy in %)**"},{"metadata":{"trusted":true},"cell_type":"code","source":"shooting_accuracy_df = df.groupby(['DistRightBound'], as_index = False).sum()\nshooting_accuracy_df.sample(3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"labeled_shooting_accuracy_df = df.groupby(['DistRightBound', 'LABEL'], as_index = False).sum()\nlabeled_shooting_accuracy_df.sample(3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Calculate field-goal percentage (goal accuracy in %):\n\nfor df in [shooting_accuracy_df, labeled_shooting_accuracy_df]:\n df['GoalsScored'] = df['made2'] + df['made3']\n df['GoalAccuracy'] = df['GoalsScored'] / df['EventID']\n\nshooting_accuracy_df.sample(3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Make final image:"},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"import matplotlib.ticker as mtick\n\nfig = plt.figure(figsize = (14, 20))\n\n\n### Plot 1 ###\n\nax1 = fig.add_axes([0, 0.4, 0.8, 0.29]) # [left, bottom, width, height]\n\nsns.lineplot(x='DistRightBound', y='GoalAccuracy', data=shooting_accuracy_df, color='crimson', ax=ax1)\nax1.set_xlabel(\"Shot distance in meters\")\nax1.set_ylabel(\"Shooting accuracy in %\")\n\n# Y ticks ar percentage:\nax1.yaxis.set_major_formatter(mtick.PercentFormatter(1.0))\n\n# Show 3 point line:\nax1.axvline(x=6.3246, color='grey', linestyle='--') # 20 feet, 9 inches\nax1.annotate('three-point line',\n xy=(6.37, 0.7), xycoords='data',\n xytext=(-100, -50), textcoords='offset points')\n\n# Add secondary axis to also show feet distance:\ndef m2feet(x):\n return x * 3.28084\ndef feet2m(x):\n return x / 3.28084\nsecax = ax1.secondary_xaxis('top', functions=(m2feet, feet2m))\nsecax.set_xlabel('Shot distance in feet')\n\nax1.set_title(\"Figure 3. Field-goal shooting accuracy % by distance,\\n2015-2019 NCAA® tournaments.\\n\")\n\n\n### Plot 2 ###\n\nax2 = fig.add_axes([0, 0, 0.8, 0.29]) # [left, bottom, width, height]\n\n# Show background image:\nax2.imshow(line_img, extent=[0, 100, 0, 100], aspect='auto', alpha=0.5)\n\n\nsns.scatterplot(x='X', y='Y', data=court_labeled_tourney_MEvents[court_labeled_tourney_MEvents.EventType == 'miss2'],\n alpha=0.35,\n edgecolor=None,\n color=colors[1], ax=ax2).set(xlim=(0, 100), ylim=(0, 100))\n\nsns.scatterplot(x='X', y='Y', data=court_labeled_tourney_MEvents[court_labeled_tourney_MEvents.EventType == 'made2'],\n alpha=0.35,\n edgecolor=None,\n color=colors[0], ax=ax2).set(xlim=(0, 100), ylim=(0, 100))\n\nsns.scatterplot(x='X', y='Y', data=court_labeled_tourney_MEvents[court_labeled_tourney_MEvents.EventType == 'miss3'],\n alpha=0.35,\n edgecolor=None,\n color=colors[1], ax=ax2).set(xlim=(0, 100), ylim=(0, 100))\n\nsns.scatterplot(x='X', y='Y', data=court_labeled_tourney_MEvents[court_labeled_tourney_MEvents.EventType == 'made3'],\n alpha=0.35,\n edgecolor=None,\n color=colors[0], ax=ax2).set(xlim=(0, 100), ylim=(0, 100))\n\nax2.legend(handles=[(Line2D([0],[0], marker='o', markerfacecolor=colors[0], \n linestyle='none', markersize=10, markeredgecolor='none')),\n (Line2D([0],[0], marker='o', markerfacecolor=colors[1],\n linestyle='none', markersize=10, markeredgecolor='none'))],\n labels=[\"goal made\", \"goal missed\"], loc=\"upper center\")\n\n\n# Remove coordinate values:\nax2.get_xaxis().set_visible(False)\nax2.get_yaxis().set_visible(False)\n\nax2.set_title(\"\\nFigure 4. Field goal accuracy by player location,\\n2015-2019 NCAA® tournaments.\\n\")\n\nsave_plot()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### What is shooting accuracy by distance per team category?"},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.lineplot(x='DistRightBound', y='GoalAccuracy', data=labeled_shooting_accuracy_df, hue=\"LABEL\", hue_order=order)\nplt.xlabel(\"Shot distance in meters\")\nplt.ylabel(\"Shooting accuracy in %\")\n\nax = plt.gca()\nhandles, labels = ax.get_legend_handles_labels()\nplt.legend(handles=handles[1:], labels=labels[1:])\n\n# Y ticks ar percentage:\nax.yaxis.set_major_formatter(mtick.PercentFormatter(1.0))\n\n# Show 3 point line:\nplt.axvline(x=6.3246, color='grey', linestyle='--') # 20 feet, 9 inches\nplt.gca().annotate('three-point line',\n xy=(6.37, 0.7), xycoords='data',\n xytext=(-100, -50), textcoords='offset points')\n\n# Add secondary axis to also show feet distance:\ndef m2feet(x):\n return x * 3.28084\ndef feet2m(x):\n return x / 3.28084\nsecax = plt.gca().secondary_xaxis('top', functions=(m2feet, feet2m))\nsecax.set_xlabel('Shot distance in feet')\n\nplt.title(\"Figure 12. Field-goal shooting accuracy % by distance per team category,\\n2015-2019 NCAA® tournaments.\\n\")\n\n\nsave_plot()\nplt.show()\n\n\nprint(\"Descriptive numbers for file nr. {} (Cinderella):\".format(str(file_nr-1)))\n\ncinderella = labeled_shooting_accuracy_df[labeled_shooting_accuracy_df.LABEL == 'Cinderella']\n\nprint('\\nGoalAccuracy < 20%:')\nprint(cinderella[cinderella.GoalAccuracy < 0.2][['DistRightBound', 'GoalAccuracy']])\n\nprint('\\nGoalAccuracy > 60%:')\nprint(cinderella[cinderella.GoalAccuracy > 0.6][['DistRightBound', 'GoalAccuracy']])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"---\n\n## 2.5. Individual Statistics, 2015-2020\n\n**Data Section 5 file: MPlayers.csv** - this file lists player ID, player's last and first names and the TeamID of the player's team [[1]](#References)."},{"metadata":{"trusted":true},"cell_type":"code","source":"MPlayers_file = \"/kaggle/input/march-madness-analytics-2020/MPlayByPlay_Stage2/MPlayers.csv\"\n\nif sys.executable != '/opt/conda/bin/python':\n # remove the forward slash if running this notebook locally:\n MPlayers_file = MPlayers_file[1:] \n \nMPlayers = pd.read_csv(MPlayers_file)\nprint(\"Num rows: {}\".format(len(MPlayers)))\nprint(\"NaN values: {}\".format(MPlayers.isna().sum().sum()))\nprint(\"Duplicated rows: {}\".format(MEvents2019.duplicated().sum()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.concat([MPlayers.head(3), MPlayers.tail(2)])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Add player's last and first names to our labeled tourney MEvents data**"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(len(labeled_tourney_MEvents))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"list(MPlayers)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"MPlayers.rename({'PlayerID': 'EventPlayerID',\n 'TeamID': 'EventTeamID'}, axis=1, inplace=True)\n\ncols = ['EventPlayerID', 'EventTeamID']\n\nplayers_labeled_tourney_MEvents = labeled_tourney_MEvents.join(MPlayers.set_index(cols), on=cols)\n\nassert len(players_labeled_tourney_MEvents) == len(labeled_tourney_MEvents)\n\nplayers_labeled_tourney_MEvents","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**One-hot encode event type**"},{"metadata":{"trusted":true},"cell_type":"code","source":"dummies = pd.get_dummies(players_labeled_tourney_MEvents['EventType']) # what type of event\n\nassert len(dummies) == len(players_labeled_tourney_MEvents)\n\nplayers_labeled_tourney_MEvents = pd.concat([players_labeled_tourney_MEvents, dummies], axis=1)\nplayers_labeled_tourney_MEvents","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Calculate offensive efficiency per season**\n\nOffensive efficiency = **(FGM + A) / (FGA - OREB + A + TO)** [[7]](#References)\n\n1. Create columns with **offensive** rebounds and **defensive** rebounds (based on EventSubType):"},{"metadata":{"trusted":true},"cell_type":"code","source":"players_labeled_tourney_MEvents['OREB'] = 0 # new column, default value\nplayers_labeled_tourney_MEvents.loc[players_labeled_tourney_MEvents.EventSubType.isin(['off', 'offdb']),\n 'OREB'] = 1\n\nplayers_labeled_tourney_MEvents['DREB'] = 0 # new column, default value\nplayers_labeled_tourney_MEvents.loc[players_labeled_tourney_MEvents.EventSubType.isin(['def', 'defdb']),\n 'DREB'] = 1\n\nplayers_labeled_tourney_MEvents","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"2. Create columns with **field goals made** and **field goals attempted**:"},{"metadata":{"trusted":true},"cell_type":"code","source":"players_labeled_tourney_MEvents['FGM'] = 0 # new column, default value\nplayers_labeled_tourney_MEvents['FGA'] = 0 # new column, default value\n\nplayers_labeled_tourney_MEvents.loc[players_labeled_tourney_MEvents.EventType.isin(['made2', 'made3']),\n 'FGM'] = 1\n\nplayers_labeled_tourney_MEvents.loc[players_labeled_tourney_MEvents.EventType.isin(['made2', 'made3', 'miss2', 'miss3']),\n 'FGA'] = 1\n\nplayers_labeled_tourney_MEvents","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"3. Group **by game and player**, sum up the events:\n\nThis step can me skipped for OE, but we will be using this data later."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Prepare column names:\n\ndummy_cols = list(dummies)\n\ngame_cols = ['Season', 'DayNum', 'WTeamID', 'LTeamID'] # to identify each game\n\ncols = ['LABEL',\n 'EventPlayerID',\n 'LastName',\n 'FirstName'] + dummy_cols + game_cols + ['OREB', 'DREB', 'FGM', 'FGA']\n\nprint(cols)","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true},"cell_type":"code","source":"sum_per_game_tourney_MEvents = players_labeled_tourney_MEvents.groupby(game_cols + ['LABEL',\n 'EventPlayerID',\n 'LastName',\n 'FirstName'],\n as_index=False).sum()[cols]\n\nprint(\"Play-by-play event logs (tournaments grouped by game and player, sum events):\")\nsum_per_game_tourney_MEvents","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"4. Group **by season and player**, sum up the events:"},{"metadata":{"trusted":true},"cell_type":"code","source":"players_season = players_labeled_tourney_MEvents.groupby(['Season',\n 'LABEL',\n 'EventPlayerID',\n 'LastName',\n 'FirstName'], as_index=False).sum()\n\nprint(\"Play-by-play event logs (tournaments grouped by season and player, sum events):\")\nplayers_season","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"5. Finally, calculate the OE:\n\nNote that we removed division by zero and negative values."},{"metadata":{"trusted":true},"cell_type":"code","source":"players_season['OE_numerator'] = players_season['FGM'] + \\\n players_season['assist']\nplayers_season['OE_denominator'] = players_season['FGA'] - \\\n players_season['OREB'] + \\\n players_season['assist'] + \\\n players_season['turnover']\n\n# Remove division by zero and negative values:\nplayers_season = players_season[(players_season['OE_denominator'] > 0)]\n\n# Calculate the OE:\nplayers_season['OE'] = (players_season['OE_numerator'] / \\\n players_season['OE_denominator']).round(2)\n\nplayers_season","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"players_season.OE.describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"6. Delete misleading results\n\nWe noticed that the formula that we are using can produce high Offensive Efficiency values even for low performing players, for example if both OE numerator and OE denominator will be \"1\", the output will be \"1\", which can lead to misleading interpretations.\n\nConsidering that we are only interested in plotting **top OE players**, we can fix this by applying filters based on the scores used in OE formula. We will eliminate all FGM scores below median."},{"metadata":{"trusted":true},"cell_type":"code","source":"players_season.FGM.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Eliminate lower 50% of summary FGM per season:\n\nprint(len(players_season))\n\nfgm_median = players_season['FGM'].median()\nplayers_season = players_season[(players_season['FGM'] > fgm_median)]\n\nprint(len(players_season))\nplayers_season.OE.describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Are Cinderella team's players more effective in offense?"},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.lineplot(x='Season', y='OE', data = players_season,\n hue='LABEL', hue_order=order, ci=False).set(ylim=(0.45, None))\n\nplt.xlabel(\"Season\")\nplt.ylabel(f'Offensive Efficiency\\n *players with at least {int(fgm_median) + 1} field goals per season')\n\nax = plt.gca()\nhandles, labels = ax.get_legend_handles_labels()\nplt.legend(handles=handles[1:], labels=labels[1:])\n\nplt.xticks(np.arange(2015, 2020, 1.0)) # custom x ticks\n\nplt.title(\"Figure 15. Mean player's* Offensive Efficiency by season per team category,\\nNCAA® tournaments.\\n\")\n\nplt.show()\n\nprint(\"Descriptive statistics for file nr. {}:\".format(str(file_nr-1)))\nplayers_season.groupby(\"LABEL\")[['OE']].describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Cinderella teams had a lowest mean player's Offensive Efficiency in season 2016 (team Syracuse and team Gonzaga) and highest in 2019 (team Oregon).\n- Offensive leaders are changing from season to season and there is no visible trend to superiority of one particular team category over another in terms of Offensive Efficiency.\n- All team categories have the same median player's Offensive Efficiency accoss last 5 seasons - 0.56.\n\nThis plot will not be included in the [Results](#III.-Results) section."},{"metadata":{},"cell_type":"markdown","source":"### Offensive leaders"},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.patches as patches\nfrom matplotlib.offsetbox import (OffsetImage, AnnotationBbox)\n\nplt.figure(figsize=(14,5))\n\ndf = players_season.groupby(['EventPlayerID',\n 'FirstName',\n 'LastName'], as_index=False).mean().sort_values('OE', ascending=False)\n\n\nsns.barplot(y=df[\"FirstName\"][:5] + \" \" + df[\"LastName\"][:5],\n x='OE', data = df[:5], color=\"#0A6FAC\", orient='h')\n\nplt.xlabel(\"Mean Offensive Efficiency by season\")\nplt.title(\"Figure 8. Top offensive players in 5 years,\\n2015-2019 NCAA® tournaments.\\n\")\n\nsave_plot()\nplt.show()\n\n\n# Some players have equal names (e.g. Chris Harris Jr.) so we need to see the team list:\nfor index, row in df[:5].iterrows():\n # from this df:\n player_id = df['EventPlayerID'][index]\n firstname = df['FirstName'][index]\n lastname = df['LastName'][index]\n \n # from external dfs:\n team_id = MPlayers.loc[MPlayers['EventPlayerID'] == player_id, 'EventTeamID'].values[0]\n team_name = MTeams.loc[MTeams['TeamID'] == team_id, 'TeamName'].values[0]\n \n print(\"{} {}, team {}\".format(firstname,\n lastname,\n team_name))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Mean per game player stats\n\nNote that we decided not to include following plots to the [Results](#III.-Results) section as we found them not enough innovative, but we will keep them in our study for the reader's independent exploration.\n\n**Calculate mean per game stats**\n\nWe already have summary events per game, now we will group by player and calculate mean:"},{"metadata":{"trusted":true},"cell_type":"code","source":"mean_per_game = sum_per_game_tourney_MEvents.groupby(['EventPlayerID',\n 'FirstName',\n 'LastName'], as_index=False).mean()\n\nmean_per_game","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(14,3.5))\n\ndf = mean_per_game.sort_values('FGM', ascending=False)\n\n\nsns.barplot(y=df[\"FirstName\"][:7] + \" \" + df[\"LastName\"][:7],\n x='FGM', data = df[:7], color=colors[1], orient='h')\n\nplt.xlabel(\"Mean field goals made\")\nplt.title(\"Field goals per game: top players,\\n2015-2019 NCAA® tournaments.\\n\")\n\nplt.show()\n\n\n# Some players have equal names (e.g. Chris Harris Jr.) so we need to see the team list: \nfor index, row in df[:7].iterrows():\n # from this df:\n player_id = df['EventPlayerID'][index]\n firstname = df['FirstName'][index]\n lastname = df['LastName'][index]\n \n # from external dfs:\n team_id = MPlayers.loc[MPlayers['EventPlayerID'] == player_id, 'EventTeamID'].values[0]\n team_name = MTeams.loc[MTeams['TeamID'] == team_id, 'TeamName'].values[0]\n \n print(\"{} {}, team {}\".format(firstname,\n lastname,\n team_name))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(14,6))\n\ndf = mean_per_game.sort_values('made3', ascending=False)\n\n\nsns.barplot(y=df[\"FirstName\"][:12] + \" \" + df[\"LastName\"][:12],\n x='made3', data = df[:12], color=colors[1], orient='h')\n\nplt.xlabel(\"Mean 3-point field goals made\")\nplt.title(\"3-point field goals per game: top players,\\n2015-2019 NCAA® tournaments.\\n\")\n\nplt.show()\n\n\n# Some players have equal names (e.g. Chris Harris Jr.) so we need to see the team list: \nfor index, row in df[:10].iterrows():\n # from this df:\n player_id = df['EventPlayerID'][index]\n firstname = df['FirstName'][index]\n lastname = df['LastName'][index]\n \n # from external dfs:\n team_id = MPlayers.loc[MPlayers['EventPlayerID'] == player_id, 'EventTeamID'].values[0]\n team_name = MTeams.loc[MTeams['TeamID'] == team_id, 'TeamName'].values[0]\n \n print(\"{} {}, team {}\".format(firstname,\n lastname,\n team_name))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(14,3.5))\n\ndf = mean_per_game.sort_values('made1', ascending=False)\n\n\nsns.barplot(y=df[\"FirstName\"][:7] + \" \" + df[\"LastName\"][:7],\n x='made1', data = df[:7], color=colors[1], orient='h')\n\nplt.xlabel(\"Mean free throws made\")\nplt.title(\"Free throws per game: top players,\\n2015-2019 NCAA® tournaments.\\n\")\n\nplt.show()\n\n\n# Some players have equal names (e.g. Chris Harris Jr.) so we need to see the team list: \nfor index, row in df[:7].iterrows():\n # from this df:\n player_id = df['EventPlayerID'][index]\n firstname = df['FirstName'][index]\n lastname = df['LastName'][index]\n \n # from external dfs:\n team_id = MPlayers.loc[MPlayers['EventPlayerID'] == player_id, 'EventTeamID'].values[0]\n team_name = MTeams.loc[MTeams['TeamID'] == team_id, 'TeamName'].values[0]\n \n print(\"{} {}, team {}\".format(firstname,\n lastname,\n team_name))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(14,3.5))\n\ndf = mean_per_game.sort_values('assist', ascending=False)\n\n\nsns.barplot(y=df[\"FirstName\"][:7] + \" \" + df[\"LastName\"][:7],\n x='assist', data = df[:7], color=colors[1], orient='h')\n\nplt.xlabel(\"Mean assists\")\nplt.title(\"Assists per game: top players,\\n2015-2019 NCAA® tournaments.\\n\")\n\nplt.show()\n\n\n# Some players have equal names (e.g. Chris Harris Jr.) so we need to see the team list: \nfor index, row in df[:7].iterrows():\n # from this df:\n player_id = df['EventPlayerID'][index]\n firstname = df['FirstName'][index]\n lastname = df['LastName'][index]\n \n # from external dfs:\n team_id = MPlayers.loc[MPlayers['EventPlayerID'] == player_id, 'EventTeamID'].values[0]\n team_name = MTeams.loc[MTeams['TeamID'] == team_id, 'TeamName'].values[0]\n \n print(\"{} {}, team {}\".format(firstname,\n lastname,\n team_name))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Defensive leaders"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(14,3.5))\n\ndf = mean_per_game.sort_values('DREB', ascending=False)\n\n\nsns.barplot(y=df[\"FirstName\"][:7] + \" \" + df[\"LastName\"][:7],\n x='DREB', data = df[:7], color=colors[0], orient='h')\n\nplt.xlabel(\"Mean defensive rebounds\")\nplt.title(\"Defensive rebounds per game: top players,\\n2015-2019 NCAA® tournaments.\\n\")\n\nplt.show()\n\n\n# Some players have equal names (e.g. Chris Harris Jr.) so we need to see the team list: \nfor index, row in df[:7].iterrows():\n # from this df:\n player_id = df['EventPlayerID'][index]\n firstname = df['FirstName'][index]\n lastname = df['LastName'][index]\n \n # from external dfs:\n team_id = MPlayers.loc[MPlayers['EventPlayerID'] == player_id, 'EventTeamID'].values[0]\n team_name = MTeams.loc[MTeams['TeamID'] == team_id, 'TeamName'].values[0]\n \n print(\"{} {}, team {}\".format(firstname,\n lastname,\n team_name))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(14,3.5))\n\ndf = mean_per_game.sort_values('block', ascending=False)\n\n\nsns.barplot(y=df[\"FirstName\"][:7] + \" \" + df[\"LastName\"][:7],\n x='block', data = df[:7], color=colors[0], orient='h')\n\nplt.xlabel(\"Mean blocks\")\nplt.title(\"Blocks per game: top players,\\n2015-2019 NCAA® tournaments.\\n\")\n\nplt.show()\n\n# Some players have equal names (e.g. Chris Harris Jr.) so we need to see the team list: \nfor index, row in df[:7].iterrows():\n # from this df:\n player_id = df['EventPlayerID'][index]\n firstname = df['FirstName'][index]\n lastname = df['LastName'][index]\n \n # from external dfs:\n team_id = MPlayers.loc[MPlayers['EventPlayerID'] == player_id, 'EventTeamID'].values[0]\n team_name = MTeams.loc[MTeams['TeamID'] == team_id, 'TeamName'].values[0]\n \n print(\"{} {}, team {}\".format(firstname,\n lastname,\n team_name))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(14,3.5))\n\ndf = mean_per_game.sort_values('steal', ascending=False)\n\n\nsns.barplot(y=df[\"FirstName\"][:7] + \" \" + df[\"LastName\"][:7],\n x='steal', data = df[:7], color=colors[0], orient='h')\n\nplt.xlabel(\"Mean steals\")\nplt.title(\"Steals per game: top players,\\n2015-2019 NCAA® tournaments.\\n\")\n\nplt.show()\n\n# Some players have equal names (e.g. Chris Harris Jr.) so we need to see the team list: \nfor index, row in df[:5].iterrows():\n # from this df:\n player_id = df['EventPlayerID'][index]\n firstname = df['FirstName'][index]\n lastname = df['LastName'][index]\n \n # from external dfs:\n team_id = MPlayers.loc[MPlayers['EventPlayerID'] == player_id, 'EventTeamID'].values[0]\n team_name = MTeams.loc[MTeams['TeamID'] == team_id, 'TeamName'].values[0]\n \n print(\"{} {}, team {}\".format(firstname,\n lastname,\n team_name))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"---\n#### Add team categories to our MEvents dataframe"},{"metadata":{"trusted":true},"cell_type":"code","source":"# First add regular season and tournaments together:\nlabeled_CompactResults = pd.concat([labeled_MRegularSeasonCompactResults, labeled_MNCAATourneyCompactResults],\n ignore_index=True)\n\ncols = ['Season', 'DayNum', 'WTeamID', 'LTeamID']\n# Next, add this data to our MEvents:\nlabeled_MEvents = MEvents.join(labeled_CompactResults.set_index(cols),\n on=cols,\n how='inner')\n\nlabeled_MEvents","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"labeled_MEvents['EventSubType'] = labeled_MEvents['EventSubType'].fillna(labeled_MEvents['EventType'])\nlabeled_MEvents = labeled_MEvents[labeled_MEvents.EventSubType != \"unk\"]\n\nlabeled_MEvents['EventType'].replace({\"made1\": \"free throw made\",\n \"miss1\": \"free throw missed\"}, inplace=True) \n\nlabeled_MEvents['EventSubType'].replace({\"1of1\": \"1 of 1\",\n \"1of2\": \"1 of 2\",\n \"2of2\": \"2 of 2\",\n \"1of3\": \"1 of 3\",\n \"2of3\": \"2 of 3\",\n \"3of3\": \"3 of 3\"}, inplace=True) \n\nlabeled_MEvents.sample(3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### What is the foul and turnover structure for Cinderella teams?"},{"metadata":{"trusted":true},"cell_type":"code","source":"labeled_MEvents[labeled_MEvents.EventType.isin(['turnover', 'foul'])].LABEL.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = go.Figure()\n\nparent_col = 'EventType'\nchild_col = 'EventSubType'\nvalue_col = 'EventID'\n\ni=0\nfor label in order:\n \n center_label = label\n \n df = labeled_MEvents[labeled_MEvents.LABEL == label]\n df = df[df.EventType.isin(['turnover', 'foul'])]\n df = df.groupby([parent_col, child_col], as_index=False).count()[[parent_col, child_col, value_col]]\n\n # We need unique ids for repeated labels:\n child_ids = list(df[parent_col] + \" - \" + df[child_col]) \n \n # Calculate values for parents:\n parent_sums = [df[value_col].sum()] # first value is the sum of all rows\n for parent in list(df[parent_col].unique()): # for each parent\n parent_sums.append(df[df[parent_col] == parent][value_col].sum()) # add sum values \n \n # Show final chart:\n fig.add_trace(go.Sunburst(\n ids = [center_label] + list(df[parent_col].unique()) + child_ids,\n labels = [center_label] + list(df[parent_col].unique()) + list(df[child_col]),\n parents = [\"\"] + [center_label]*df[parent_col].nunique() + list(df[parent_col]),\n values = parent_sums + list(df[value_col]),\n textinfo='label+percent parent',\n branchvalues=\"total\",\n domain=dict(column=i)))\n \n i+=1\n\nfig.update_layout(\n grid= dict(columns=3, rows=1),\n margin = dict(t=0, l=0, r=0, b=0),\n uniformtext=dict(minsize=10, mode='hide')\n)\n\nfig.update_layout(\n annotations=[\n dict(\n x=0.5,\n y=0,\n showarrow=False,\n align=\"center\",\n text=\"pers - personal foul | off - offensive foul
\\\nbpass - bad pass turnover | lostb - lost ball | offen - offensive turnover | \\\ntrav - travelling | other - other type of turnover\",\n xref=\"paper\",\n yref=\"paper\",\n font=dict(size=14),\n )\n ])\n\n\nfig.update_layout(\n colorway=[\"#17344B\",\"#D485AF\"],\n title={\n 'text': \"Fouls and turnovers by subtype per team category,
2015-2019. Interactive graph.\",\n 'y':0.9,\n 'x':0.5,\n 'xanchor': 'center',\n 'yanchor': 'top'},\n font=dict(\n family='sans-serif',\n color=\"#000\")\n)\n\n\nfig.show(renderer=\"kaggle\")\nfig_9 = go.Figure(fig) # to show the same fig in the Results section","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### What is the structure of free throw attempts for Cinderella teams?"},{"metadata":{"trusted":true},"cell_type":"code","source":"labeled_MEvents[labeled_MEvents.EventType.isin(['free throw made', 'free throw missed'])].LABEL.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = go.Figure()\n\nparent_col = 'EventType'\nchild_col = 'EventSubType'\nvalue_col = 'EventID'\n\ni=0\nfor label in order:\n \n center_label = label\n \n df = labeled_MEvents[labeled_MEvents.LABEL == label]\n df = df[df.EventType.isin(['free throw made', 'free throw missed'])]\n df = df.groupby([parent_col, child_col], as_index=False).count()[[parent_col, child_col, value_col]]\n\n # We need unique ids for repeated labels:\n child_ids = list(df[parent_col] + \" - \" + df[child_col]) \n \n # Calculate values for parents:\n parent_sums = [df[value_col].sum()] # first value is the sum of all rows\n for parent in list(df[parent_col].unique()): # for each parent\n parent_sums.append(df[df[parent_col] == parent][value_col].sum()) # add sum values \n \n # Show final chart:\n fig.add_trace(go.Sunburst(\n ids = [center_label] + list(df[parent_col].unique()) + child_ids,\n labels = [center_label] + list(df[parent_col].unique()) + list(df[child_col]),\n parents = [\"\"] + [center_label]*df[parent_col].nunique() + list(df[parent_col]),\n values = parent_sums + list(df[value_col]),\n textinfo='label+percent parent',\n branchvalues=\"total\",\n domain=dict(column=i)))\n \n i+=1\n \n\nfig.update_layout(\n grid= dict(columns=3, rows=1),\n margin = dict(t=0, l=0, r=0, b=0),\n uniformtext=dict(minsize=10, mode='hide')\n)\n\n\nfig.update_layout(\n colorway=[\"#0173B2\",\"#DC143C\"],\n title={\n 'text': \"Free throw attempts per team category,
2015-2019. Interactive graph.\",\n 'y':0.9,\n 'x':0.5,\n 'xanchor': 'center',\n 'yanchor': 'top'},\n font=dict(\n family='sans-serif',\n color=\"#000\")\n)\n\n\nfig.show(renderer=\"kaggle\")\nfig_4 = go.Figure(fig) # to show the same fig in the Results section\n\n\nseasons = list(labeled_MEvents['Season'].unique())\n\ndf = labeled_double_MRegularSeasonDetailedResults[labeled_double_MRegularSeasonDetailedResults.Season.isin(seasons)]\ncinderella_vs_ordinary(df, \"played\", \"regular season of 2015-2019\", \"FTA\")\ncinderella_vs_ordinary(df, \"played\", \"regular season of 2015-2019\", \"FTM\")\n\ndf = labeled_double_MNCAATourneyDetailedResults[labeled_double_MNCAATourneyDetailedResults.Season.isin(seasons)]\ncinderella_vs_ordinary(df, \"played\", \"tournaments of 2015-2019\", \"FTA\")\ncinderella_vs_ordinary(df, \"played\", \"tournaments of 2015-2019\", \"FTM\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"---\n\n## 2.6. Public Rankings, 2003-2020\n\n**Data Section 4 file: MMasseyOrdinals.csv** - this file lists out rankings (e.g. 1, 2, 3, ..., N) of teams going back to the 2002-2003 season, under a large number of different ranking system methodologies. By convention, the final pre-tournament rankings are always expressed as RankingDayNum=133, even though sometimes the rankings for individual systems are not released until Tuesday (DayNum=134) or even Wednesday or Thursday [[1]](#References)."},{"metadata":{"trusted":true},"cell_type":"code","source":"MMasseyOrdinals = None\nMMasseyOrdinals = load_file(MMasseyOrdinals, 'MMasseyOrdinals')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Filter out tournament teams**\n\nOnly include rows if both season and team ID is present in the tourney data:"},{"metadata":{"trusted":true},"cell_type":"code","source":"MMasseyOrdinals = MMasseyOrdinals[(MMasseyOrdinals['Season'].isin(MNCAATourneyCompactResults['Season']) & \n (MMasseyOrdinals['TeamID'].isin(MNCAATourneyCompactResults['WTeamID'])))]\nMMasseyOrdinals","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Add the labels - Ordinary, Cinderella and Top**"},{"metadata":{"trusted":true},"cell_type":"code","source":"cols = ['Season', 'TeamID']\n\nlabeled_MMasseyOrdinals = MMasseyOrdinals.join(cinderellas.set_index(cols), on=cols)\nlabeled_MMasseyOrdinals = labeled_MMasseyOrdinals.join(top_seeded.set_index(cols), on=cols)\n\n# Create a categorical LABEL column:\nlabel = labeled_MMasseyOrdinals[['Cinderella', 'Top']]\nlabel = pd.DataFrame(label.idxmax(1))\nlabeled_MMasseyOrdinals['LABEL'] = label\n\n# Fill in the missing values:\nlabeled_MMasseyOrdinals['LABEL'] = labeled_MMasseyOrdinals['LABEL'].fillna(\"Ordinary\")\n\n# Fill in the missing values:\nlabeled_MMasseyOrdinals['Cinderella'] = labeled_MMasseyOrdinals['Cinderella'].fillna(0) # not a cinderella\nlabeled_MMasseyOrdinals['Top'] = labeled_MMasseyOrdinals['Top'].fillna(0) # not a top\n\nassert len(labeled_MMasseyOrdinals) == len(MMasseyOrdinals)\n\nlabeled_MMasseyOrdinals","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"How many rows per each category?"},{"metadata":{"trusted":true},"cell_type":"code","source":"labeled_MMasseyOrdinals.LABEL.value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### What is the mean ranking for Cinderella teams?"},{"metadata":{"trusted":true},"cell_type":"code","source":"df = labeled_MMasseyOrdinals\n\n# Make plot:\nsns.barplot(df['RankingDayNum'], df['OrdinalRank'], hue=df['LABEL'],\n hue_order=order, dodge=False, errwidth=1.5, alpha=0.75)\n\nplt.legend(loc='upper center', bbox_to_anchor=(0.5, 1.075), ncol=3, fancybox=True)\n\nplt.xlabel(\"Day no. (of regular season)\")\nplt.ylabel(\"Mean overall ranking\")\n\n# Fewer x ticks:\nfor tick_label in plt.gca().xaxis.get_ticklabels()[::2]:\n tick_label.set_visible(False)\n \nplt.xticks(rotation=90)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Some bars look odd, checking why:"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(labeled_MMasseyOrdinals[labeled_MMasseyOrdinals.RankingDayNum == 20].LABEL.value_counts())\nprint(\"\\n\")\nprint(labeled_MMasseyOrdinals[labeled_MMasseyOrdinals.RankingDayNum == 20].SystemName.value_counts())\nprint(\"\\n\")\nprint(labeled_MMasseyOrdinals[labeled_MMasseyOrdinals.RankingDayNum == 20].OrdinalRank.describe())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(labeled_MMasseyOrdinals[labeled_MMasseyOrdinals.RankingDayNum == 96].LABEL.value_counts())\nprint(\"\\n\")\nprint(labeled_MMasseyOrdinals[labeled_MMasseyOrdinals.RankingDayNum == 96].SystemName.value_counts())\nprint(\"\\n\")\nprint(labeled_MMasseyOrdinals[labeled_MMasseyOrdinals.RankingDayNum == 96].OrdinalRank.describe())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"labeled_MMasseyOrdinals.SystemName.nunique()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def ranking_comparison(df, lower, upper):\n '''Compare Cinderella team rankings vs. other team categories'''\n\n df_top = df[df.LABEL == 'Top']\n df_cinderella = df[df.LABEL == 'Cinderella']\n df_ordinary = df[df.LABEL == 'Ordinary']\n\n total_top_games = len(df_top)\n total_cinderella_games = len(df_cinderella)\n total_ordinary_games = len(df_ordinary)\n\n\n between_medians_top = len(df_top[(df_top.OrdinalRank > lower) &\n (df_top.OrdinalRank < upper)])\n\n between_medians_cinderella = len(df_cinderella[(df_cinderella.OrdinalRank > lower) &\n (df_cinderella.OrdinalRank < upper)])\n\n between_medians_ordinary = len(df_ordinary[(df_ordinary.OrdinalRank > lower) &\n (df_ordinary.OrdinalRank < upper)])\n\n share = between_medians_cinderella/total_cinderella_games\n share_top = between_medians_top/total_top_games\n share_ordinary = between_medians_ordinary/total_ordinary_games\n\n if share > 0.51:\n share_str = '{:.0%}'.format(share)\n share_top_str = '{:.0%}'.format(share_top)\n share_ordinary_str = '{:.0%}'.format(share_ordinary)\n\n return share_str, share_top_str, share_ordinary_str","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Final figure to include in the [Results](#III.-Results):"},{"metadata":{"trusted":true},"cell_type":"code","source":"df = labeled_MMasseyOrdinals[~labeled_MMasseyOrdinals.SystemName.isin([\"DES\", \"BIH\"])]\nprint(f'{df.Season.min()}-{df.Season.max()}')\n\nsystem_cnt = df.SystemName.nunique()\n\ndf = df.groupby(['LABEL', 'RankingDayNum'], as_index=False).mean()\n\ni=0\nfor label in order:\n\n plot_df = df[df.LABEL == label].sort_values(by='RankingDayNum')\n\n plt.plot(plot_df['RankingDayNum'], plot_df['OrdinalRank'])\n plt.fill_between(plot_df['RankingDayNum'], plot_df['OrdinalRank'], color=label_colors[i],\n alpha=0.25, label=label)\n i+=1\n\nplt.ylim(0, 200)\nplt.gca().margins(0)\n\nplt.legend(loc='upper center', bbox_to_anchor=(0.5, 1.075), ncol=3, fancybox=True)\n\nplt.xticks(np.arange(1, 134, 3))\nplt.xticks(rotation=90)\n\nplt.xlabel(\"Day no. (of regular season)\")\n\nplt.ylabel(\"BEST\" + \" \"*30 + \"Mean overall ranking\" + \" \"*30 + \"WORST\")\n\nplt.title(\"Figure 13. Team category vs. pre-tournament ranking\\nacross {} ranking systems, 2003-2019.\".format(system_cnt), y=1.1)\n\nsave_plot()\nplt.show()\n\n\nprint(\"Descriptive statistics for file nr. {}:\\n\".format(str(file_nr-1)))\nfor label in order:\n print(\"{}: median rank: {}, mean: {}.\".format(label, int(df[df.LABEL == label]['OrdinalRank'].median()),\n round(df[df.LABEL == label]['OrdinalRank'].mean(), 2)))\n\nshare_str, share_top_str, share_ordinary_str = ranking_comparison(labeled_MMasseyOrdinals, 20, 80) \nprint(f'\\nIn {share_str} public rankings (of 172 ranking systems) in 2003-2019,'\n f' Cinderella teams were ranked between 20'\n f' and 80 vs. {share_top_str} for the Top and {share_ordinary_str} for Ordinary teams.')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"---\n**Narrow down to 5 popular rating systems** - Pomeroy (POM), Sagarin (SAG), RPI (RPI), ESPN BPI (EBP) and ESPN SOR (ESR)\n\nThe description of each system will be included in the [Results](#III.-Results) section."},{"metadata":{"trusted":true},"cell_type":"code","source":"labeled_MMasseyOrdinals_five = labeled_MMasseyOrdinals[labeled_MMasseyOrdinals['SystemName'].isin(['POM', 'SAG', 'RPI', 'EBP', 'ESR'])]\nlabeled_MMasseyOrdinals_five.SystemName.value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### How are the rankings distributed per each rating system?"},{"metadata":{"trusted":true},"cell_type":"code","source":"subplot_titles=['Pomeroy', 'RPI', 'Sagarin', 'ESPN BPI', 'ESPN SOR']\n\nfig = make_subplots(rows=5, cols=1,\n shared_xaxes=True, subplot_titles=subplot_titles, vertical_spacing = 0.05)\n\nrow = 1 # row nr. for subplot\nfor system_name in ['POM', 'SAG', 'RPI', 'EBP', 'ESR']: # Make plots for each rating system:\n\n i = 0\n for label in order: # 'Ordinary', 'Cinderella', 'Top'\n \n df = labeled_MMasseyOrdinals_five[labeled_MMasseyOrdinals_five.LABEL == label]\n print(f'{df.Season.min()}-{df.Season.max()}' + \" \"*100, end=\"\\r\", flush=True)\n df = df[df.SystemName == system_name]\n \n fig.add_trace(\n go.Box(x=df['OrdinalRank'],\n name=label,\n marker_color=sns.color_palette(\"colorblind\").as_hex()[i],\n boxmean=True, # represent mean\n boxpoints=\"suspectedoutliers\",\n visible=True), \n row=row, col=1)\n \n i+=1\n \n row+=1 # go to next subplot\n\n \nfig.update_layout(showlegend=False, # hide ledend\n width=plotly_width, height=800) # set size\n\n# Set axis font:\nfig.update_yaxes(tickfont=dict(size=14))\n\n# Add titles:\nfig.update_xaxes(title_text='Overall ranking from best to worst', row=5, col=1)\n\n\n# Plot title:\nfig.update_layout(\n title={\n 'text': \"Team category vs. pre-tournament ranking distribution,
2003-2019. Interactive graph.\",\n 'y':0.95,\n 'x':0.5,\n 'xanchor': 'center',\n 'yanchor': 'top'},\n font=dict(\n family='sans-serif',\n color=\"#000\"),\n margin=dict(t=120), # margin between title and plot\n boxgap=0.35,\n boxgroupgap=0\n)\n\n\nfig.show(renderer=\"kaggle\")\nfig_11 = go.Figure(fig) # to show the same fig in the Results section\n\n\nshare_str, share_top_str, share_ordinary_str = ranking_comparison(labeled_MMasseyOrdinals_five, 20, 80) \n\nprint(f'In {share_str} public rankings of Pomeroy, RPI, Sagarin, ESPN BPI and ESPN SOR in 2003-2019,'\n ' Cinderella teams were ranked between 20'\n f' and 80 vs. {share_top_str} for the Top and {share_ordinary_str} for Ordinary teams.')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### What is the average pre-tournament ranking per each season?\n\nFilter out final pre-tournament rankings released before the play-in games:"},{"metadata":{"scrolled":false,"trusted":true},"cell_type":"code","source":"df = labeled_MMasseyOrdinals_five[labeled_MMasseyOrdinals_five['RankingDayNum'] == 133]\nprint(f'{df.Season.min()}-{df.Season.max()}')\n\n# plt.figure(figsize=(14,14))\nsns.lineplot(x=\"Season\", y=\"OrdinalRank\", data=df, hue='LABEL',\n style='SystemName', hue_order=order, ci=None).set(xlim=(2003, 2019))\n\nax = plt.gca()\nlegend = ax.legend()\nlegend.texts[0].set_text(\"Team category\")\nlegend.texts[4].set_text(\"\\nRanking system\")\n\nax.invert_yaxis() # to show best rating on top\n\nplt.xlabel(\"Season\")\nplt.ylabel(\"WORST\" + \" \"*32 + \"Overall ranking\" + \" \"*32 + \"BEST\")\n\nplt.title(\"Figure 14. Final pre-tournament rankings by season vs. team category\\n\")\n\nsave_plot()\nplt.show()\n\n\nshare_str, share_top_str, share_ordinary_str = ranking_comparison(labeled_MMasseyOrdinals_five, 20, 65) \n\nprint(f'In {share_str} of final pre-tournament rankings in 2003-2019,'\n ' Cinderella teams were ranked between 20'\n f' and 65 vs. {share_top_str} for the Top and {share_ordinary_str} for Ordinary teams.')\n\ndf[df.LABEL == 'Cinderella'].groupby(['Season', 'SystemName'])['OrdinalRank'].describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Note how in a season of 2013 rankings were more spread out from the average. In other words, this season has the biggest standard deviation from average (EBP: 52.16, POM: 44.43, RPI: 26.85, SAG: 43.02) of the pre-tournament rankings among 3 Cinderella teams - FL Gulf Coast, La Salle and Oregon."},{"metadata":{},"cell_type":"markdown","source":"---\n\n## 2.7. Prediction Experiment\n\nConsidering the above analysis, we will try to predict which team could become a Cinderella if the 2020 tournament would not be canceled.\n\n### Data preprocessing\n\nWe believe that rankings are important for \"Cinderellaness\", so we will build our input data based on available rankings. This will also mean that we will not use any data before season 2003."},{"metadata":{"trusted":true},"cell_type":"code","source":"MMasseyOrdinals = load_file(MMasseyOrdinals, 'MMasseyOrdinals')\n\nMMasseyOrdinals.SystemName.value_counts().head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It is important to have **as many data samples as possible**, so we will use 3 ranking systems that occur most frequently in a data:"},{"metadata":{"trusted":true},"cell_type":"code","source":"ml_MMasseyOrdinals = MMasseyOrdinals[MMasseyOrdinals['SystemName'].isin(['SAG', 'MOR', 'POM'])]\nml_MMasseyOrdinals","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ml_MMasseyOrdinals.SystemName.value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Calculate mean ranking per each system**"},{"metadata":{"trusted":true},"cell_type":"code","source":"ml_MMasseyOrdinals = ml_MMasseyOrdinals.groupby(['Season', 'TeamID', 'SystemName'], as_index=False).mean()\nml_MMasseyOrdinals = ml_MMasseyOrdinals.drop('RankingDayNum', 1)\nml_MMasseyOrdinals","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"One-hot encode mean rankings:"},{"metadata":{"trusted":true},"cell_type":"code","source":"dummies = pd.get_dummies(ml_MMasseyOrdinals['SystemName']) \n\nfor col in dummies:\n ml_MMasseyOrdinals[col] = dummies[col]*ml_MMasseyOrdinals['OrdinalRank'] \n\nml_MMasseyOrdinals = ml_MMasseyOrdinals.groupby(['Season', 'TeamID'], as_index=False).sum()\nml_MMasseyOrdinals = ml_MMasseyOrdinals.drop('OrdinalRank', 1)\n\nml_MMasseyOrdinals","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Prepare regular season data\n\nSelect the same seasons as we have in rankings:"},{"metadata":{"trusted":true},"cell_type":"code","source":"seasons = list(ml_MMasseyOrdinals.Season.unique())\nml_double_MRegularSeason = double_MRegularSeasonDetailedResults[double_MRegularSeasonDetailedResults.Season.isin(seasons)]\nml_double_MRegularSeason","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Calculate mean metrics:"},{"metadata":{"trusted":true},"cell_type":"code","source":"ml_double_MRegularSeason = ml_double_MRegularSeason.groupby(['Season', 'TeamID'], as_index=False).mean()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Add output label: Cinderella**"},{"metadata":{"trusted":true},"cell_type":"code","source":"cols = ['Season', 'TeamID']\nml_double_MRegularSeason = ml_double_MRegularSeason.join(cinderellas.set_index(cols), on=cols)\n\n# Fill in the missing values:\nml_double_MRegularSeason['Cinderella'] = ml_double_MRegularSeason['Cinderella'].fillna(0)\n\nml_double_MRegularSeason","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ml_double_MRegularSeason.Cinderella.value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Feature selection\n\nWe will manually select features that we believe contribute most to the \"Cinderellaness\".\n\nWe already chose 3 ranking systems and calculated mean rankings. Now we will select which columns to keep from the regular season data. Basically we would like remove metrics like field goals attempted (including two-point field goals and three-point field goals), opponent rebound columns and simply irrelevant columns like day number and winning / losing team IDs."},{"metadata":{"trusted":true},"cell_type":"code","source":"ml_double_MRegularSeason = ml_double_MRegularSeason.drop(['FGA', 'FGA2', 'FGA3',\n 'DayNum', 'WTeamID', 'LTeamID',\n 'OppOR', 'OppDR'], 1)\n\nml_double_MRegularSeason","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Join two data files together"},{"metadata":{"trusted":true},"cell_type":"code","source":"cols = ['Season', 'TeamID']\nml_data = ml_double_MRegularSeason.join(ml_MMasseyOrdinals.set_index(cols),\n on=cols,\n how='inner').reset_index(drop=True)\n\nml_data","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(list(ml_data))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Move 2020 data to a separate dataframe"},{"metadata":{"trusted":true},"cell_type":"code","source":"ml_data_2020 = ml_data[ml_data.Season == 2020]\nml_data_2020 = ml_data_2020.drop('Cinderella', 1)\nml_data_2020","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ml_data = ml_data[ml_data.Season != 2020]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ml_data.Cinderella.value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Note that **our data is imbalanced - only 34 Cinderella cases vs. 5799 non-Cinderella cases**. This could be a potential problem for a classification model. We will address this later."},{"metadata":{},"cell_type":"markdown","source":"Prepare X (input), y (output) data for a machine learning:"},{"metadata":{"trusted":true},"cell_type":"code","source":"X = ml_data.loc[:, ml_data.columns != 'Cinderella']\ny = ml_data[['Cinderella']]\n\nprint(X.shape, y.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Split data into random train and test subsets\n\nWe will leave 25% of data as a test set that our model will not use for training."},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0)\n\nprint(X_train.shape, y_train.shape)\nprint(X_test.shape, y_test.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_train.Cinderella.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_test.Cinderella.value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Baseline model\n\nConsidering that our data is imbalanced, we will use a DummyClassifier model as a baseline. The model will simply use 'most_frequent' strategy and predict the most frequent label (non-Cinderella)."},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.dummy import DummyClassifier\nfrom sklearn.metrics import accuracy_score\n\nclf_dummy = DummyClassifier(strategy='most_frequent', random_state=0).fit(X_train, y_train)\ny_pred = clf_dummy.predict(X_test) \n\nprint(\"Dummy model accuracy (most frequent label): %0.2f\" % (accuracy_score(y_test, y_pred)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Note how we got 99% accuracy score on a dummy model, so in our case **it is not a good metric** to evaluate the real model. We will use metrics that are **more suitable for imbalanced datasets**: [balanced accuracy score](https://scikit-learn.org/stable/modules/model_evaluation.html#balanced-accuracy-score), [F1 score](https://scikit-learn.org/stable/modules/model_evaluation.html#precision-recall-f-measure-metrics) and [ROC AUC](https://scikit-learn.org/stable/modules/model_evaluation.html#roc-metrics)."},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import cross_validate\n\nscoring = ['balanced_accuracy', 'f1_macro', 'roc_auc']\n\nscores = cross_validate(clf_dummy, X_train, y_train, cv=5, scoring=scoring)\nsorted(scores.keys())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"scores['test_roc_auc']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def print_scores(scores):\n '''Print out classification metrics'''\n \n print(\"Balanced accuracy: %0.2f (+/- %0.2f)\" % (scores['test_balanced_accuracy'].mean(), scores['test_balanced_accuracy'].std() * 2))\n print(\"F1 score: %0.2f (+/- %0.2f)\" % (scores['test_f1_macro'].mean(), scores['test_f1_macro'].std() * 2))\n print(\"ROC AUC: %0.2f (+/- %0.2f)\" % (scores['test_roc_auc'].mean(), scores['test_roc_auc'].std() * 2))\n\nprint(\"Baseline model scores:\\n\")\nprint_scores(scores)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Benchmark\n\nNow when we have our dummy baseline model, we will test out different models and check which one gives better results. Our goal is to train a model that will have at least **0.60 F1 score and 0.70 ROC AUC on a test data it had never seen**.\n\nConsidering historical data, we expect that **there could be 0 to 5 Cinderella teams in 2020** (if the tournament would not be canceled).\n\nAs a matter of fact, 1 to 3 Cinderella teams per season is what we would expect the most:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Cinderellas per season:\nseason_team_cinderellas.groupby('Season').count()['Cinderella'].describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Classifier comparison\n\nWe will train several models and evaluate the results. We will use **cross-validation instead of a train / test data**, because our dataset is small and we want to **maximize the number of samples** which can be used for learning the model. We will leave test dataset untouched for now and use only train dataset for both model training and cross-validation.\n\n#### Support Vector Classification\n\nSupport vector machines (SVMs) are a set of supervised learning methods used for classification, regression and outliers detection [[27]](#References)."},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn import svm\n\nclf = svm.SVC(kernel='rbf')\n\n# Evaluating estimator performance:\nscores = cross_validate(clf, X_train, y_train, cv=5, scoring=scoring)\nprint_scores(scores)\n\nclf.fit(X_train, y_train) # training the model\npred = clf.predict(ml_data_2020) # predicting an output\n\nprint(f'\\nCinderellas in 2020: {np.count_nonzero(pred == 1)}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Support Vector Classification, Balanced\n\nThe \"balanced\" mode uses the values of y to automatically adjust weights inversely proportional to class frequencies in the input data [[27]](#References)."},{"metadata":{"trusted":true},"cell_type":"code","source":"clf = svm.SVC(kernel='rbf', class_weight='balanced')\n\n# Evaluating estimator performance:\nscores = cross_validate(clf, X_train, y_train, cv=5, scoring=scoring)\nprint_scores(scores)\n\nclf.fit(X_train, y_train) # training the model\npred = clf.predict(ml_data_2020) # predicting an output\n\nprint(f'\\nCinderellas in 2020: {np.count_nonzero(pred == 1)}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Random Forest Classification\n\nA random forest is a meta estimator that fits a number of decision tree classifiers on various sub-samples of the dataset and uses averaging to improve the predictive accuracy and control over-fitting [[28]](#References)."},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.ensemble import RandomForestClassifier\n\nclf = RandomForestClassifier()\n\n# Evaluating estimator performance:\nscores = cross_validate(clf, X_train, y_train, cv=5, scoring=scoring)\nprint_scores(scores)\n\nclf.fit(X_train, y_train) # training the model\npred = clf.predict(ml_data_2020) # predicting an output\n\nprint(f'\\nCinderellas in 2020: {np.count_nonzero(pred == 1)}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### XGBoost classification\n\nXGBoost is an optimized distributed gradient boosting library designed to be highly efficient, flexible and portable. It implements machine learning algorithms under the Gradient Boosting framework. XGBoost provides a parallel tree boosting (also known as GBDT, GBM) that solve many data science problems in a fast and accurate way [[29]](#References)."},{"metadata":{"trusted":true},"cell_type":"code","source":"from xgboost import XGBClassifier\n\nclf = XGBClassifier()\n\n# Evaluating estimator performance:\nscores = cross_validate(clf, X_train, y_train, cv=5, scoring=scoring)\nprint_scores(scores)\n\nclf.fit(X_train, y_train) # training the model\npred = clf.predict(ml_data_2020) # predicting an output\n\nprint(f'\\nCinderellas in 2020: {np.count_nonzero(pred == 1)}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Initial results\n\nOut of 4 different models that we have trained, one (Support Vector Classification, Balanced) had a balanced accuracy (0.83) greater than our dummy baseline model (0.50). Unfortunately, the same model **had a lowest F1 score (0.43)** of all four and has predicted 97 teams to become a Cinderellas. \n\nA major source of limitation is due to imbalanced data. To address this issue, we will perform over-sampling using SMOTE - Synthetic Minority Over-sampling Technique.\n\n### Refinement: over-sampling using SMOTE\n\nPerform over-sampling using SMOTE:"},{"metadata":{"trusted":true},"cell_type":"code","source":"from imblearn.over_sampling import SMOTE \n\nsm = SMOTE(random_state=42)\nX_train_resampled, y_train_resampled = sm.fit_resample(X_train, y_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_train_resampled.Cinderella.value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"After over-sampling is done, our **new data** has 4348 Cinderella cases and 4348 non-Cinderella cases.\n\n#### Support Vector Classification\n\nNote that we will not use the \"balanced\" model now, because we already improved data balance by over-sampling."},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.metrics import f1_score\nfrom sklearn.metrics import roc_auc_score\n\nclf = svm.SVC(kernel='rbf')\n\n# Evaluating estimator performance:\nscores = cross_validate(clf, X_train_resampled, y_train_resampled, cv=5, scoring=scoring)\nprint_scores(scores)\n\nclf.fit(X_train_resampled, y_train_resampled) # training the model\n\ny_pred = clf.predict(X_test) # using the model on a test set\npred = clf.predict(ml_data_2020) # predicting an output\n\n# Evaluating performance on a new data:\nprint(\"\\nF1 score (test data): %0.2f\" % (f1_score(y_test, y_pred, average='macro')))\nprint(\"ROC AUC (test data): %0.2f\" % (roc_auc_score(y_test, y_pred)))\n \nprint(f'\\nCinderellas in 2020: {np.count_nonzero(pred == 1)}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Random Forest Classification"},{"metadata":{"trusted":true},"cell_type":"code","source":"clf = RandomForestClassifier()\n\n# Evaluating estimator performance:\nscores = cross_validate(clf, X_train_resampled, y_train_resampled, cv=5, scoring=scoring)\nprint_scores(scores)\n\nclf.fit(X_train_resampled, y_train_resampled) # training the model\n\ny_pred = clf.predict(X_test) # using the model on a test set\npred = clf.predict(ml_data_2020) # predicting an output\n\n# Evaluating performance on a new data:\nprint(\"\\nF1 score (test data): %0.2f\" % (f1_score(y_test, y_pred, average='macro')))\nprint(\"ROC AUC (test data): %0.2f\" % (roc_auc_score(y_test, y_pred)))\n \nprint(f'\\nCinderellas in 2020: {np.count_nonzero(pred == 1)}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### XGBoost classification"},{"metadata":{"trusted":true},"cell_type":"code","source":"clf = XGBClassifier()\n\n# Evaluating estimator performance:\nscores = cross_validate(clf, X_train_resampled, y_train_resampled, cv=5, scoring=scoring)\nprint_scores(scores)\n\nclf.fit(X_train_resampled, y_train_resampled) # training the model\n\ny_pred_XGB = clf.predict(X_test) # using the model on a test set\npred = clf.predict(ml_data_2020) # predicting an output\n\n# Evaluating performance on a new data:\nprint(\"\\nF1 score (test data): %0.2f\" % (f1_score(y_test, y_pred_XGB, average='macro')))\nprint(\"ROC AUC (test data): %0.2f\" % (roc_auc_score(y_test, y_pred_XGB)))\n\nprint(f'\\nCinderellas in 2020: {np.count_nonzero(pred == 1)}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Refinement results\n\nAfter over-sampling, all three models have improved results on a new training data.\n\nSVC model resulted with average balanced accuracy of 0.85, F1 score of 0.85 and ROC AUC of 0.86, but didn't perform well on a test data, having a low F1 score of 0.45.\n\nRandom Forest model had balanced accuracy and F1 score of 0.99 and ROC AUC of 1.00 in cross validation. On a test data, it showed F1 score of 0.56 and ROC AUC of 0.62.\n\n**XGBoost model performed best** and showed the same cross validation metrics as Random Forest model, but **better results with the test data** - 0.61 F1 score and 0.68 ROC AUC."},{"metadata":{},"cell_type":"markdown","source":"### Selected model hyperparameter tuning\n\nWe will try to improve the XGBoost model performance by changing some hyperparameters (parameter descriptions are from [[30]](#References))."},{"metadata":{"trusted":true},"cell_type":"code","source":"clf.get_params() # current model parameters","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**'eta'** - Step size shrinkage used in update to prevents overfitting. After each boosting step, we can directly get the weights of new features, and eta shrinks the feature weights to make the boosting process more conservative."},{"metadata":{"trusted":true},"cell_type":"code","source":"scores = []\nfor eta in np.arange(0, 1, 0.15):\n clf = XGBClassifier(eta=eta).fit(X_train_resampled, y_train_resampled)\n y_pred = clf.predict(X_test) # using the model on a test set\n score = f1_score(y_test, y_pred, average='macro')\n scores.append(score)\n print(\"eta: {:.2f} / \".format(eta) + \"F1 score (test data): %0.2f\" % (score))\n\nplt.figure(figsize = (7.5,3))\nplt.xlabel('eta')\nplt.ylabel('F1 score')\nplt.plot(np.arange(0, 1, 0.15), scores)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**'max_depth'** - Maximum depth of a tree. Increasing this value will make the model more complex and more likely to overfit. 0 is only accepted in lossguided growing policy when tree_method is set as hist and it indicates no limit on depth."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"scores = []\nfor max_depth in range(1, 10):\n clf = XGBClassifier(max_depth=max_depth).fit(X_train_resampled, y_train_resampled)\n y_pred = clf.predict(X_test) # using the model on a test set\n score = f1_score(y_test, y_pred, average='macro')\n scores.append(score)\n print(\"max_depth: {:.2f} / \".format(max_depth) + \"F1 score (test data): %0.2f\" % (score))\n\nplt.figure(figsize = (7.5,3))\nplt.xlabel('max_depth')\nplt.ylabel('F1 score')\nplt.plot(range(1, 10), scores)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**'subsample'** - Subsample ratio of the training instances. Setting it to 0.5 means that XGBoost would randomly sample half of the training data prior to growing trees. and this will prevent overfitting. Subsampling will occur once in every boosting iteration."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"scores = []\nfor subsample in np.arange(0, 1, 0.10):\n clf = XGBClassifier(subsample=subsample).fit(X_train_resampled, y_train_resampled)\n y_pred = clf.predict(X_test) # using the model on a test set\n score = f1_score(y_test, y_pred, average='macro')\n scores.append(score)\n print(\"subsample: {:.2f} / \".format(subsample) + \"F1 score (test data): %0.2f\" % (score))\n\nplt.figure(figsize = (7.5,3))\nplt.xlabel('subsample')\nplt.ylabel('F1 score')\nplt.plot(np.arange(0, 1, 0.10), scores)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**'n_estimators'** – Number of gradient boosted trees. Equivalent to number of boosting rounds."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"scores = []\nfor n_estimators in [50, 100, 200, 250, 500]:\n clf = XGBClassifier(n_estimators=n_estimators).fit(X_train_resampled, y_train_resampled)\n y_pred = clf.predict(X_test) # using the model on a test set\n score = f1_score(y_test, y_pred, average='macro')\n scores.append(score)\n print(\"n_estimators: {:.2f} / \".format(n_estimators) + \"F1 score (test data): %0.2f\" % (score))\n\nplt.figure(figsize = (7.5,3))\nplt.xlabel('n_estimators')\nplt.ylabel('F1 score')\nplt.plot([50, 100, 200, 250, 500], scores)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Using the final model\n\nTrain the best model and make final predictions:"},{"metadata":{"trusted":true},"cell_type":"code","source":"model = XGBClassifier(subsample = 0.2, n_estimators = 200)\n\nmodel.fit(X_train_resampled, y_train_resampled) # training the model\n\ny_pred = model.predict(X_test) # using the model on a test set\n\n\n# Evaluating performance on a new data:\nprint(\"ROC AUC (test data): %0.2f\\n\" % (roc_auc_score(y_test, y_pred)))\n \npred = model.predict(ml_data_2020) # predicting an output\nprint(f'Cinderellas in 2020: {np.count_nonzero(pred == 1)}')\n\npred_proba = model.predict_proba(ml_data_2020) # also get probabilities","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Compare updated model with the previous one:**"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.metrics import classification_report\nprint(\"\\nXGBoost:\")\nprint(classification_report(y_test, y_pred_XGB))\nprint(\"\\nXGBoost (tuned parameters):\")\nprint(classification_report(y_test, y_pred))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Show a confusion matrix"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.metrics import confusion_matrix\n\nfig, ax = plt.subplots(1,2, figsize = (12,5))\n\ncm = confusion_matrix(y_test, y_pred_XGB)\n\ncm_labels = ['Other', 'Cinderella']\n\nsns.heatmap(cm,\n cmap=sns.cubehelix_palette(),\n cbar=False,\n annot=True, annot_kws={\"size\": 13.5}, fmt='g',\n xticklabels=cm_labels,\n yticklabels=cm_labels, ax=ax[0])\n\nax[0].set_title(\"XGBoost\\nF1 score: 0.61\\n\")\n\nax[0].set_xlabel(\"\\nPredicted label\")\nax[0].set_ylabel(\"True label\\n\")\n\n\ncm = confusion_matrix(y_test, y_pred)\n\nsns.heatmap(cm,\n cmap=sns.cubehelix_palette(),\n cbar=False,\n annot=True, annot_kws={\"size\": 13.5}, fmt='g',\n xticklabels=cm_labels,\n yticklabels=cm_labels, ax=ax[1])\n\nax[1].set_title(\"XGBoost (tuned parameters)\\nF1 score: 0.63\\n\")\n\nax[1].set_xlabel(\"\\nPredicted label\")\nax[1].set_ylabel(\"True label\\n\")\n\nplt.subplots_adjust(wspace=0.35)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Determine the way floating point numbers, arrays and other NumPy objects are displayed:\nnp.set_printoptions(formatter={'float_kind':'{:f}'.format})\n\n# Each probability will be formatted as so:\npred_proba[0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Find out which teams are predicted to become a Cinderellas**"},{"metadata":{"trusted":true},"cell_type":"code","source":"ml_data_2020['Cinderella'] = pred\nml_data_2020['Probability'] = np.split(pred_proba, 2, 1)[1]\npred_cinderellas = ml_data_2020[ml_data_2020.Cinderella == 1][['Season', 'TeamID', 'Probability']]\npred_cinderellas","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Team names:"},{"metadata":{"trusted":true},"cell_type":"code","source":"MTeams[MTeams.TeamID.isin(pred_cinderellas.TeamID)]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Seed numbers from previous seasons, just an additional check:"},{"metadata":{"trusted":true},"cell_type":"code","source":"for team in pred_cinderellas.TeamID:\n print(f'Team: {MTeams.loc[MTeams.TeamID == team, \"TeamName\"].values[0]}')\n print(MNCAATourneySeeds[MNCAATourneySeeds.TeamID == team].sort_values('Season', ascending=False).head())\n print('\\n')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Based on our experiment, 6 teams were the potential candidates to become a Cinderellas: **Arizona St, ETSU, Florida, Illinois, Indiana and Providence**."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Show saved images in the \"Results\" section:\n\noutput_dir = '/kaggle/working/'\n\nif sys.executable != '/opt/conda/bin/python':\n # remove the forward slash if running this notebook locally:\n output_dir = output_dir[1:]\n\ndef display_img(filename):\n if os.path.isfile(output_dir + filename):\n display(Image(output_dir + filename))\n else:\n print(\"Image not found. Re-run this cell when the Implementation section is executed!\")\n\nfig_error = \"Graph not found. Re-run this cell when the Implementation section is executed!\"\ndef display_fig(fig):\n fig.show(renderer=\"kaggle\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# III. Results\n\n## 3.1. General Findings\n\nThis section summarises general findings about men's NCAA® basketball across different seasons from 1985 to 2020. \n\nAll **calculations** for the numbers mentioned in our findings are available in the [Implementation](#II.-Implementation) section."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"display_img(\"07.png\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Based on a 2010-2020 geo data, Philadelphia is the \"capital\" of men's NCAA® basketball, being a home city of 6 tournament participant teams - Drexel, La Salle, Pennsylvania, St Joseph's, Temple and Villanova. Upon further research, we have verified that 5 of these teams (La Salle, Pennsylvania, St. Joseph’s, Temple and Villanova) form a [Philadelphia Big 5](http://www.philadelphiabig5.org/) - an informal association of college athletic programs."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"try:\n display_fig(fig_1)\nexcept NameError:\n print(fig_error)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Most field goals are scored in the first 5 minutes of the second half, in all seasons from 2015 to 2019. Our interpretation is that probably after the first half is over, a team knows an opponent's moves and can adjust game tactics accordingly. Another reason for this could be that players can have more energy after a rest time, pumped up and motivated to continue a competitive game."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"display_img(\"09.png\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Based on a 2015-2019 data:\n\n- Highest 2-point shot accuracy is achieved in the basket area. Accuracy is lowest at about 3 meters from the basket for two-point goals and there is an interesting peak at about 5 meters where the accuracy improves a little bit.\n\n- For the best 3-point shot accuracy player should make a shot from just behind the three-point line. Shots made from about 9.5-meter distance are more accurate comparing to those made from 8-meter distance."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"display_img(\"08.png\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- For the data available with court coordinates (2019-2020), about 69% of the three-point goals were made from either \"outside right\" or \"outside left\" area, and only 21% were made from the \"outside center\" area.\n\n- Could it be because of a possibly stronger defense in the center area? While our data is lacking X, Y coordinates for defensive events like rebounds, blocks and steals, we can see on the second figure that 41% of the turnovers beyond the three-point line happened just about there in the center. There are many actions that can result in a turnover, including: ball stolen by opposing team, throwing a bad pass, throwing the ball out of bounds, stepping out of bounds, committing a double-dribble, palming or traveling violation, committing a backcourt violation, shot clock violation, three-second violation, five-second violation or an offensive foul (charge or illegal screen) [[5]](#References).\n\n- In addition, we assume that if more data would be available, the goals could be distributed more evenly along the three-point line."},{"metadata":{"scrolled":false,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"display_img(\"01.png\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"*Note.* Each point in a scatter plot represents an observation in the dataset. In this figure we are looking at a relationship between game location (for the winner team) and points scored. We see that dark points (opponent floor games) are lower and tend a little bit towards the right side (less points for visitor, more points for home team) and pink points (home games) are higher and tend slightly towards left (again, more points for home team, less points for visitor).\n\nBased on a 1985-2020 data:\n\n- Number of points scored by the winning team (a scoring margin, also called \"margin of victory\") will be higher at home games and lower in road.\n- Opponent floor games are more competitive and will end with a lower scoring margin if a visiting team manages to win. A popular explanation is that it might be hard to dominate the opponent on his territory."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"display_img(\"02.png\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"*Note.* In this graph we plot mean scoring margin along with a 95% confidence interval for that mean (a range of values that we can be 95% certain contains the mean). \n\nBased on a 1985-2020 data:\n\n- The common pattern that we can observe in all seasons from 1985 to 2020 is that, as we saw before, home games are won with highest scoring margin (all season mean 13.63), opponent floor games - with lowest scoring margin (all season mean 9.46), and neutral court games are somewhere in between (all season mean 11.09). It is also worth mentioning that game location is important when evaluating team selection for participation in the tourney - starting with the 2018 season, a team's schedule and results are broken down in four quadrants that place greater emphasis on games played on neutral courts and in true road environments [[6]](#References)."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"display_img(\"11.png\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"*Note.* To calculate Offensive Efficiency we used the following formula: (FGM + A) / (FGA - OREB + A + TO) [[7]](#References)\n\nBased on a 2015-2019 data, top 5 offensive players in five years using Offensive Efficiency metric by season would be:\n\n1. **Norense Odiase, team Texas Tech**. Odiase completed his Texas Tech career as the winningest player in program history having been a part of teams that compiled 108 victories [[8]](#References).\n2. **Gorjok Gak, team Florida**. Rated a three-star prospect by ESPN, Rivals and 247 Sports. Scored 86 points and grabbed 96 rebounds in 46 career appearances [[9]](#References).\n3. **Jalen Smith, team Maryland**. Jalen Smith was named a third-team all-American by the Associated Press, adding to his list of end-of-season accolades. Smith also earned third-team honors from CBS Sports, Sports Illustrated and Sporting News [[10]](#References).\n4. **Tyrique Jones, team Xavier**. Jones is third in the nation with 20 double-doubles in latest season. He leads the Big East in rebounding (11.1) and he's Xavier's second-leading scorer (13.7). His field goal percentage (54.5) is the best on the team and fifth in the league [[11]](#References).\n5. **Mitchell Solomon, team Oklahoma St**. Earned honorable mention All-Big 12 Conference honors from the Associated Press. In Senior Season (2018), led the Oklahoma State Cowboys in field goal percentage (minimum 50 attempts), making 54.5 percent of his shots. Also led the Cowboys in rebounds per game with 6.5 rebounds per game with nearly half of those coming as offensive rebounds [[12]](#References)."},{"metadata":{},"cell_type":"markdown","source":"## 3.2. Uncovering Cinderellaness\n\nIn this section we demonstrate our findings about a key features that define a Cinderella team compared to other team categories. "},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"display_img(\"03.png\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- A season of 1999 had the most Cinderella\n teams. As we saw before in the [Implementation](#II.-Implementation) section, these teams are: Gonzaga, Miami OH, Missouri St, Oklahoma and Purdue.\n- The only season when there were no Cinderella teams is a season of 1995."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"try:\n display_fig(fig_2)\nexcept NameError:\n print(fig_error)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Based on a 1985-2019 data:\n\n- No. 1 seeds have won the most championships and most games in any particular round.\n- None of the teams seeded Nos. 9 through 16 have won the national semifinals nor the national final (metaphorically speaking, this phenomenon is referred to as Cinderella's chariot turning back into a pumpkin).\n- None of No. 16 seeds have ever become a Cinderella or even won in a Round 2.\n- No. 11 seed Cinderella teams have actually performed better in Elite 8 comparing to No. 10 seeds, having won 4 games vs. only 1 for the 10 seed Cinderella."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"display_img(\"04.png\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"*Note.* A box plot (or box-and-whisker plot) shows the distribution of quantitative data in a way that facilitates comparisons between variables or across levels of a categorical variable. The box shows the quartiles of the dataset while the whiskers extend to show the rest of the distribution, except for points that are determined to be outliers [[13]](#References).\n\nBased on a 1985-2019 data:\n\n- Our results demonstrated that in 58% of games won in regular season, Cinderella teams had a scoring margin above 10 (mean: 14.62, median: 12.0) vs. 47% of games for the Ordinary teams (mean: 11.76, median: 10.0).\n- With more competition in the NCAA® tournaments, it is harder for a Cinderella team to keep up the high scoring margin, so the mean value in tournaments is even lower than for the Ordinary teams (Cinderella: 8.89, Ordinary: 9.91). If we look at the median values of this metric they are actually the same (Cinderella: 8, Ordinary: 8).\n- In both regular season and tournaments, Top teams match expectations and keep higher scoring margin that both previously mentioned categories. As discussed before, we expect to see the same pattern in all similar plots."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"display_img(\"05.png\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Based on a 1985-2019 data:\n\n- A scoring margin tend to decrease from round to round for both Ordinary\n and Top\n category winner teams. In our interpretation, the scoring margin shows how much power and influence a winner team has over a losing team. With competitiveness increasing as the national final is getting closer, such an influence gets weaker.\n- For Cinderella\n teams though, mean scoring margin is on it's highest level (10.04) in Round 2. We speculate that this might be due to the teams being more motivated and engaged after a successful Round 1, but it gets a lot harder to score points in the Sweet 16 against stronger opponents."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"try:\n display_fig(fig_3)\nexcept NameError:\n print(fig_error)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"*Note.* In this graph and all similar graphs, use dropdown menu and / or buttons to switch between different states. \n\nBased on a 2003-2019 data:\n\n- Cinderellas are good at shooting 2-pointers in regular season, but not so much in the tournaments (compared to other teams). The opposite is true for the three-point goals.\n\n\n- In 61% of games played in regular season, Cinderella teams had more than 17 two-point field goals per game, with the mean of 18.91 vs. 17.69 2-pointers for Ordinary teams. In NCAA® tournaments the median score (19 goals) decreased to an Ordinary team level (17 goals).\n- If we look at only winning games, in regular season Cinderella teams didn't show much difference in performance compared to Ordinary (median 19 two-point field goals for both categories). As for the tournaments, in 58% of games won in tournaments, Cinderella teams had less than 19 2-point field goals, with the mean of 17.67 vs. 18.85 goals for the Ordinary teams.\n\n\n- In 56% of games played in tournaments, Cinderella teams had more than 6 three-point goals per game (mean: 6.83, median: 7.0) vs. 47% of games for the Ordinary teams (mean: 6.54, median: 6.0).\n- For games won in the tournaments, mean three-point goals per game for Cinderellas (7.31) is even higher than the same for the Top category teams (6.89). You need to make more points per game in order to win against a stronger opponent, and the three-point goal is a perfect tool for accomplishing that."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"try:\n display_fig(fig_4)\nexcept NameError:\n print(fig_error)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Based on a 2015-2019 data:\n\n- Cinderella teams have the biggest share of missed free throws (32% missed vs. 68% made) among all team categories.\n\n- Cinderellas have missed the \"first of two\" free throw attempts in 37% of attempts and have made a successful \"first of two\" shot in 42% of shots. \n\n- In 56% of games played in tournaments of 2015-2019, Cinderella teams had less than 13 free throws made (mean: 11.78, median: 12) vs. 49% of games for the Ordinary teams (mean: 12.89, median: 13).\n\n- Despite imbalanced labels (70524 free throw events for Ordinary, 650 for Cinderella and 7522 for Top teams), the structure of free throw attempts looks very similar for all three categories."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"try:\n display_fig(fig_5)\nexcept NameError:\n print(fig_error)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Based on a 2003-2019 data:\n\n- Three-point field goal ratio is lower than two-point field goal ratio for all team categories.\n- Cinderella\n teams have highest three-point field goal ratio in NCAA® tournaments of all team categories for both games played (37.7%) and games won (41.2%).\n- Cinderella teams have a little bit higher two-point field goal ratio than Ordinary\n teams in both regular season (50.5% vs. 48.2%) and NCAA® tournaments (48.1% vs. 46.9% for Ordinary).\n- In 57% of games won in regular season, Cinderella teams had more than 36 two-point field goal attempts (which is a median value for Ordinary teams)."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"try:\n display_fig(fig_6)\nexcept NameError:\n print(fig_error)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Based on a 2003-2019 data, Cinderella teams had higher Assist to Turnover Ratio than Ordinary teams in both regular season (1.11) and NCAA® tournaments (1.06). The same is true for the games won - Assist to Turnover of 1.23 in regular season and 1.17 in the tournaments. Assist to Turnover Ratio is a way to determine how often a player turns the ball over when trying to distribute. This metric reveals a lot about a team's ability to play point guard efficiently [[14]](#References)."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"try:\n display_fig(fig_7)\nexcept NameError:\n print(fig_error)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Based on a 2003-2019 data:\n\n- In 54% of games played in regular season, Cinderella teams had more than 23 defensive rebounds per game, with the mean value of 24.21 vs. 23.65 defensive rebounds for the Ordinary teams.\n- Similar pattern is true for the tournaments - in 56% games, Cinderellas had more than 22 defensive rebounds, which is a median value for the Ordinary.\n- The situation changes if we filter out only games won - in 53% of games (won) in regular season and in 54% of NCAA® tournament games Cinderella teams had less defensive rebounds than Ordinary teams.\n\n\n- In 55% of games played in regular season, Cinderella teams had more than 6 steals per game (mean: 7.18, median: 7) vs. 46% of games for the Ordinary teams (mean: 6.47, median: 6.0).\n\n\n- Cinderella teams had more than 3 blocks per game in 55% of games won in regular season, with the mean value of 4.04 vs. 3.78 blocks for the Ordinary (median: 3 blocks)."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"try:\n display_fig(fig_8)\nexcept NameError:\n print(fig_error)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Based on a 2015-2019 data:\n\n- Cinderellas\n might be able to defend without fouling - they had better blocks per personal fouls ratio than Ordinary\n teams in both regular season (21.4% vs. 17.8%) and NCAA® tournaments (21.2% vs. 16.7%).\n\n- In 59% of games played in tournaments, Cinderella teams had less than 18 personal fouls (mean: 16.72, median: 16.0) vs. 45% of games for the Ordinary teams (mean: 18.21, median: 18.0)."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"try:\n display_fig(fig_9)\nexcept NameError:\n print(fig_error)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Based on a 2015-2019 data:\n\n- Foul share of total \"foul plus turnover\" events for Cinderella (53%) is greater than the same for Ordinary teams (51%) but less than the same for Top teams (55%).\n\n- Cinderellas have almost identical turnover structure as Ordinary teams, both having equal share of bad pass turnover (14%), lost ball turnover (13%), offensive turnover (5%) and travelling turnover (4%).\n\n- Top teams have smaller personal foul share (45% vs. 48%) and greater offensive foul share (6% vs. 5%) than Cinderella teams."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"try:\n display_fig(fig_10)\nexcept NameError:\n print(fig_error)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"*Note*. To calculate Rebound Margin we used the following formula: RPG - OPP RPG [[15]](#References).\n\nBased on a 2003-2019 data:\n\n- In 60% of games played in regular season, Cinderella teams had a positive Rebound Margin (mean: 2.44, median: 2.0) vs. 47% of games for the Ordinary teams (mean: -0.17, median: 0.0).\n\n- Cinderella teams had a Rebound Margin greater than -2.0 (mean: -0.97, median: 0.0) in 54% of games played in tournaments vs. 49% of games for the Ordinary teams (mean: -1.71, median: -2.0).\n\n- If we look at the games won only, Cinderella teams have lowest Rebound Margin among all team categories (mean: 1.48, median: 2.0), so we should not credit rebounding for a Cinderella's success in game. We also have to be mindful of the fact that this metric does not take into account defensive vs. offensive rebounds."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"display_img(\"10.png\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Based on a 2015-2019 data:\n- Cinderella\n teams were not so good at shooting 2-pointers from about 3.5 meters from basket, but they were more accurate compared to other team categories when a shot from 2-meter distance was made.\n- For the three-point goals, Cinderella teams were most successful in shooting from about 8.5-meter distance.\n- Although these findings apply to our data, we believe that if more data would be available, we would not see much difference among the three team categories."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"display_img(\"12.png\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- In 81% public rankings (of 172 ranking systems) in 2003-2019, Cinderella teams were ranked between 20 and 80 (median rank: 49, mean: 53.81.) vs. 15% for the Top (median: 9, mean: 11.95) and 27% for Ordinary teams (median: 119, mean: 134.79). Public rankings could play an effective role in predicting which team is a potential Cinderella of the upcoming season."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"try:\n display_fig(fig_11)\nexcept NameError:\n print(fig_error)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Based on a 2003-2019 data:\n\n- The plots look very similar for each of the five different ranking systems. If we look closely at the mean and median rankings for Cinderella teams, we can note that RPI (mean rank 52.91, median: 47) and ESPN SOR (mean: 49.76, median: 48) ranking would be the best and Sagarin the worst (mean: 56.95, median: 52).\n- In 83% public rankings of Pomeroy, RPI, Sagarin, ESPN BPI and ESPN SOR in 2003-2019, Cinderella teams were ranked between 20 and 80 vs. 17% for the Top and 27% for Ordinary teams.\n\n
\nWhat is the difference between Pomeroy, RPI, Sagarin, ESPN BPI and ESPN SOR?\n\n**Pomeroy** - Ken Pomeroy ranking system that incorporates statistics like shooting percentage, margin of victory, and strength of schedule, ultimately calculating offensive, defensive, and overall \"efficiency\" numbers for all teams in Division I. Higher-ranked teams are predicted to beat lower-ranked teams on a neutral court [[16]](#References).\n\n**RPI** - the Rating Percentage Index (RPI) has been used by the NCAA men's basketball committee since 1981, as supplemental data to help select at-large teams and seed all teams for the men's and women's NCAA basketball tournaments. The three component factors which make up the RPI are as follows: (25%) the team's Division I winning percentage, (50%) team's opponents' Division I winning percentage, (25%) team's opponents' opponents' Division I winning percentage [[17]](#References).\n\n**Sagarin** - Jeff Sagarin rankings that aim to do the same thing as the Pomeroy ratings, but use a different formula, one that doesn't (appear to) factor in stats like shooting percentage (though the algorithm is proprietary and, thus, not entirely transparent) [[16]](#References). The overall rating is a synthesis of the three different score-based methods: PREDICTOR, GOLDEN_MEAN, and RECENT [[18]](#References).\n\n**ESPN BPI** - a predictive rating system for college basketball that's designed to measure team strength and project performance going forward. In the simplest sense, BPI (College Basketball Power Index) is a power rating that can be used to determine how much better one team is than another [[19]](#References). \n\n**ESPN SOR** - ESPN's Strength of Record takes strength of schedule a step further by accounting for how a team actually did against its schedule. Unlike BPI, which accounts for how the game was won, Strength of Record simply cares about the difficulty of a team’s schedule and the result (win or loss) [[19]](#References)."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"display_img(\"13.png\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- In 71% of final pre-tournament rankings in 2003-2019, Cinderella\n teams were ranked between 20 and 65 vs. 16% for the Top\n and 21% for Ordinary\n teams. This is consistent with what has been found in previous graphs.\n- It is notable that the ranking range for each category stays quite consistent from season to season despite that each season has it's own, different Cinderella teams."},{"metadata":{},"cell_type":"markdown","source":"## 3.3. Predicting Cinderellas\n\nWe have trained a machine learning model to predict which teams could become a Cinderellas in season 2020 if the tournament would not be canceled. \n\nOur final model used XGBoost classifier. It was **able to predict whether or not a team will be a Cinderella with a 0.98 accuracy** on a data it had never seen. Considering that input data was heavily imbalanced (only 34 Cinderella cases vs. 5799 non-Cinderella cases), we used **F1 score and ROC AUC** (area under the ROC curve) metrics to evaluate final results. \n\nWe acknowledge that \"Cinderellaness\" is a tricky feature that is not straightforward to predict, so we appreciated achieving a macro average F1 score of 0.63 and ROC AUC of 0.74 on a test dataset.\n\nAccording to our results, **top 3 potential Cinderella teams of 2020** could be:\n\n- ETSU (99.6% probability)\n- Florida (98.0% probability)\n- Providence (83.6% probability)\n\n
\n\n\t
BUCCANEERS
\n\t
GATORS
\n\t
FRIARS
\n\n
\nAlthough we do not have a ground truth data to check our predictions, we have verified that a similar assumption about **ETSU Cinderella potential** was discussed in SPORTS ILLUSTRATED [[20]](#References), NBC Sports [[21]](#References) and USA TODAY [[22]](#References).\n\nPlease refer to [Implementation](#II.-Implementation) section to see all 6 Cinderella candidates that our model has predicted."},{"metadata":{},"cell_type":"markdown","source":"---\n# References\n\n[1] Kaggle. (2020). Google Cloud & NCAA® March Madness Analytics. Data Description. [Online]. Available: https://www.kaggle.com/c/march-madness-analytics-2020/data\n
[2] NCAA. (2020). NCAA cancels remaining winter and spring championships. [Online]. Available: https://www.ncaa.org/about/resources/media-center/news/ncaa-cancels-remaining-winter-and-spring-championships\n
[3] J. Boozell. (2019). The 11 greatest March Madness Cinderella stories. [Online]. Available: https://www.ncaa.com/news/basketball-men/2019-02-21/11-greatest-march-madness-cinderella-stories\n
[4] K. Bonsor and D. Roos. (2003). How March Madness Works. [Online]. Available: https://entertainment.howstuffworks.com/march-madness.htm\n
[5] Jr NBA. (n.d.). Turnover. [Online]. Available: https://jr.nba.com/turnover/\n
[6] NCAA. (ca. 2020). Selection Criteria. [Online]. Available: http://www.ncaa.org/about/resources/media-center/mens-basketball-selections-101-selections\n
[7] Thunder StatLab. (n.d.). OFFENSIVE EFFICIENCY. [Online]. Available: https://www.nba.com/resources/static/team/v2/thunder/statlab-OE-191201.pdf\n
[8] Texas Tech University. (ca. 2020). NORENSE ODIASE. [Online]. Available: https://texastech.com/sports/mens-basketball/roster/norense-odiase/6580\n
[9] University Athletic Assoc., Inc., FOX Sports Sun & IMG College. (ca. 2019). GORJOK GAK. [Online]. Available: https://floridagators.com/sports/mens-basketball/roster/gorjok-gak/11067\n
[10] E. Giambalvo. (2020). Maryland basketball’s Jalen Smith earns third-team all-American honors. [Online]. Available: https://www.washingtonpost.com/sports/2020/03/20/maryland-basketballs-jalen-smith-earns-third-team-all-american-honors/\n
[11] R. Wilson. (2020). Can 'unbreakable' Tyrique Jones carry Xavier into NCAA Tournament? [Online]. Available: https://www.wcpo.com/sports/college-sports/xavier-university-sports/can-unbreakable-tyrique-jones-carry-xavier-into-ncaa-tournament\n
[12] Oklahoma State University Athletics. (ca. 2020). MITCHELL SOLOMON. [Online]. Available: https://okstate.com/sports/mens-basketball/roster/mitchell-solomon/4051\n
[13] Seaborn. (n.d.). seaborn.boxplot. [Online]. Available: https://seaborn.pydata.org/generated/seaborn.boxplot.html\n
[14] M. Badger. (ca. 2014). Stat Central: Understanding Strengths, Shortcomings Of Assist Rate Metrics. [Online]. Available: https://hoopshabit.com/2013/08/18/stat-central-understanding-strengths-shortcomings-of-assist-rate-metrics/\n
[15] NCAA. (2020). Men's Basketball. TEAM STATISTICS. REBOUND MARGIN. [Online]. Available: https://www.ncaa.com/stats/basketball-men/d1/current/team/151\n
[16] S. Paruk. (2020). Which Advanced Metric Should Bettors Use: KenPom or Sagarin? [Online]. Available: https://www.sportsbettingdime.com/guides/strategy/kenpom-vs-sagarin/\n
[17] Collegiate Basketball News Company. (n.d.). What is the RPI? [Online]. Available: http://rpiratings.com/WhatisRPI.php\n
[18] J. Sagarin. (2020). Jeff Sagarin's College Basketball Ratings. [Online]. Available: http://sagarin.com/sports/cbsend.htm\n
[19] ESPN Sports Analytics Team. (2016). BPI and Strength of Record: What are they and how are they derived? [Online]. Available: https://www.espn.com/blog/statsinfo/post/_/id/125994/bpi-and-strength-of-record-what-are-they-and-how-are-they-derived\n
[20] K. Sweeney. (2020). Cinderella Spotlight: Steve Forbes Has Built a Mid-Major Force at East Tennessee State. [Online]. Available: https://www.si.com/college/2020/03/11/march-madness-cinderellas-etsu-basketball\n
[21] R. Dauster. (2020). Introducing Cinderella: East Tennessee State doesn’t need an at-large bid anymore. [Online]. Available: https://collegebasketball.nbcsports.com/2020/03/09/introducing-cinderella-east-tennessee-state-doesnt-need-an-at-large-bid-anymore/\n
[22] S. Gleeson. (2020). Six mid-major teams that had potential to be Cinderella before coronavirus canceled March Madness. [Online]. Available: https://eu.usatoday.com/story/sports/ncaab/2020/03/16/coronavirus-march-madness-ncaa-tournament-cinderella-potential/5012987002/\n
[23] D. Wilco. (2020). What is March Madness: The NCAA tournament explained. [Online]. Available: https://www.ncaa.com/news/basketball-men/bracketiq/2020-04-20/what-march-madness-ncaa-tournament-explained\n
[24] City location coordinates obtained via [GeoPy Nominatim](https://geopy.readthedocs.io/en/stable/#nominatim) geocoder for OpenStreetMap data. (The MIT License). [Online]. Available: https://www.kaggle.com/evanca/ncaageocities\n
[25] Court outline image (Figures 4-6) courtesy of author.\n
[26] K. Bonsor. (2003). How Basketball Works. Scoring. [Online]. Available: https://entertainment.howstuffworks.com/basketball4.htm\n
[27] Scikit-learn. (n.d.). Support Vector Machines. [Online]. Available: https://scikit-learn.org/stable/modules/svm.html\n
[28] Scikit-learn. (n.d.). 3.2.4.3.1. sklearn.ensemble.RandomForestClassifier. [Online]. Available: https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html\n
[29] Xgboost developers. (n.d.). XGBoost Documentation. [Online]. Available: https://xgboost.readthedocs.io/en/latest/\n
[30] Xgboost developers. (n.d.). XGBoost Parameters. [Online]. Available: https://xgboost.readthedocs.io/en/latest/parameter.html\n\n\n
Use of external Open Source packages:\n
https://github.com/Phlya/adjustText (The MIT License)\n
https://github.com/nvictus/svgpath2mpl (The 3-Clause BSD License)\n\n\n\n---\n\n#### Web Accessibility Statement\nWe aimed for a visually friendly designs and used **color schemes** that should be easily identified by people with all types of color vision. We ensured that default **fonts** are no smaller than 9 points/pixels in all of our plots. Your feedback and suggestions are welcome about how we can continue to improve the accessibility.\n"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.6"}},"nbformat":4,"nbformat_minor":4}