{"cells":[{"metadata":{},"cell_type":"markdown","source":"#    **At-Large and Seeding Automation via Machine Learning:** \n<p>The NCAA Men's basketball season can be viewed in three distinct sections. The first is the regular season. This is when all non-conference and conference games are played. The second and shortest is the conference tournament where teams can earn a chance to play in the NCAA Tournament. Lastly, the third section - reserved for only 68 teams - is the tournament used to crown the champion.  It could be fairly stated the reason NCAA men's basketball teams play the regular season is for the opportunity to play in the NCAA tournament. Some teams are guaranteed this opportunity by winning their end of season conference tournament. Teams that don't win their conference tournament may still receive an invitation via an at-large selection as a result of the high quality of their regular season and conference tournament play.</p>\n<p>After the selection committee chooses the at-large teams, it must assign all teams, both conference tournament winners and at-large selections, a seed. The purpose of this notebook is two-fold: first, to model the at-large teams and second, to assign each team a seed. Both processes will use machine learning approaches to derive the results.<p/>\n<p>Along the way, we'll compare the expected versus actual results. Who made it into the at-large field under the wire and who was left standing in the cold? Who received higher or lower seeding than expected?<p/>\nWith a high degree of accuracy, can we automate the at-large and seeding process for the NCAA Men's Basketball Tournament? How do we handle the 2020 season?"},{"metadata":{},"cell_type":"markdown","source":"# Table of Contents\n\n1.  [Data Wrangling](#intLink1)\n1.  [Strength of Schedule](#intLink2)\n1.  [Pythagorean Expectation](#intLink3)\n1.  [Massey Ordinals](#intLink4)\n1.  [Aggregating the Data](#intLink5)\n1.  [Model Creation: At-Large Selections](#intLink6)\n1.  [At-Large Selection Model Results](#intLink7)\n1.  [Wrangling the 2020 Season Data](#intLink9)\n1.  [Model Creation: Team Seeding](#intLink10)\n1.  [Seeding Model Results](#intLink11)\n1.  [The Tournament We All Wanted: 2020](#intLink13)\n1.  [2020 Final Results](#intLink14)\n1.  [2020 Bids per Conference](#intLink15)\n1.  [Reviewing Model Results for 2017-2019](#intLink16)\n1.  [A Few Final Words](#intLink17) \n\n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# We'll import the files we need from Kaggle and two that have been uploaded: massey_ord_2020 and at_large_2020. These two files are necessary for our 2020 predictions. \nimport numpy as np\nimport pandas as pd \nimport lightgbm as lgb\nimport matplotlib\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport warnings\nwarnings.simplefilter(action='ignore', category=FutureWarning) \nwarnings.simplefilter(action='ignore')\nnp.random.seed(924)\n\n# Importing the files needed to perform the necessary analytics\nreg_season = pd.read_csv('/kaggle/input/march-madness-analytics-2020/MDataFiles_Stage2/MRegularSeasonCompactResults.csv')\nncaa_tourn = pd.read_csv('/kaggle/input/march-madness-analytics-2020/MDataFiles_Stage2/MNCAATourneyCompactResults.csv')\nmassey_ord = pd.read_csv('/kaggle/input/march-madness-analytics-2020/MDataFiles_Stage2/MMasseyOrdinals.csv')\nmassey_ord_2020 = pd.read_csv('/kaggle/input/ncaa-supplementals/MMasseyOrdinals_2020_only.csv')\nncaa_seeds = pd.read_csv('/kaggle/input/march-madness-analytics-2020/MDataFiles_Stage2/MNCAATourneySeeds.csv')\nconference = pd.read_csv('/kaggle/input/march-madness-analytics-2020/MDataFiles_Stage2/MTeamConferences.csv')\nconference.rename(columns={'TeamID':'Team'}, inplace = True)\nteams = pd.read_csv('/kaggle/input/march-madness-analytics-2020/MDataFiles_Stage2/MTeams.csv')\nteams.rename(columns={'TeamID':'Team'}, inplace = True)\nat_large = pd.read_csv('/kaggle/input/ncaa-supplementals/NCAA_auto_bids.csv')\nat_large_2020 = at_large[(at_large.Season ==2020)]\n\n\n# Because we'll need this function later and it doesn't fit well anywhere else, we'll create it now. \n# It makes the seeding file usable by removing the location designation \n# from the seed while converting it from a str to an int. \nseeds_df = ncaa_seeds.loc[:, ['TeamID', 'Season', 'Seed']]\n\ndef clean_seed(seed):\n    s_int = int(seed[1:3])\n    return s_int\n\nseeds_df['seed_int'] = seeds_df['Seed'].apply(lambda x: clean_seed(x))\nseeds_df.drop(labels=['Seed'], inplace=True, axis=1) # This is the string label\nseeds_df.rename(columns={'TeamID':'Team'}, inplace = True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<div id=\"intLink1\"></div>\n# Data Wrangling\nWe first need to create a data frame that shows each regular season game as two instances with each team listed as the primary and secondary (opponent) team. This makes compiling each team's statistics significantly easier. We won't include the game location and overtime inputs as they don't factor into any of our analytics."},{"metadata":{"trusted":true},"cell_type":"code","source":"def duplicate_games(reg_season):    \n    reg_season_w = reg_season.rename(columns = {'WTeamID': 'Team',\n                                             'WScore': 'Team_Score',                                                             \n                                             'LTeamID': 'Opponent',\n                                             'LScore': 'Opponent_Score'})\n\n    reg_season_l = reg_season.rename(columns = {'LTeamID': 'Team',\n                                            'LScore': 'Team_Score',                                                             \n                                            'WTeamID': 'Opponent',\n                                            'WScore': 'Opponent_Score'})\n\n    regseason = (reg_season_w, reg_season_l)\n    regseason = pd.concat(regseason, ignore_index = True, sort = False)\n    regseason = regseason[['Season','DayNum', 'Team', 'Team_Score', 'Opponent', 'Opponent_Score']]\n    return regseason\n\nreg_season = duplicate_games(reg_season)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now that we have a data frame in a usable form, the first step is to create input variables. The two problems we're automating use a lot of the same variables which isn't a surprise because there's implied overlap to the problems; the reasons to select a team are also reasons to assign a seeding. We'll create one master data frame that will be used as a base for both the at-large and seeding selection. My original data frame was approximately 150 input variables but most proved to provide no additional insight into solving these two problems. By excluding these variables, we are able to speed up the training process. A lot of the standard box score stats - steals, rebounds, etc. were tried but didn't improve model performance. "},{"metadata":{"trusted":true},"cell_type":"code","source":"# These inputs are self-explanatory. We create variables for a team’s margin of victory and a win/loss binary. \n# Additionally, we get the averages for points scored and allowed along with a team’s winning percentage.\n# One input variable that was surprisingly helpful was a team’s minimal margin of victory for a season. \n\nreg_season['mov'] = reg_season['Team_Score'] - reg_season['Opponent_Score']\nreg_season['win'] = np.where(reg_season.mov > 0,1,0)\nreg_season['avg_off'] = reg_season.groupby(['Season','Team'])['Team_Score'].transform('mean')\nreg_season['avg_def'] = reg_season.groupby(['Season','Team'])['Opponent_Score'].transform('mean')\nreg_season['wp'] = reg_season.groupby(['Season','Team'])['win'].transform('mean')\nreg_season['mov_min'] = reg_season.groupby(['Season','Team'])['mov'].transform('min')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<div id=\"intLink2\"></div>\n# Strength of Schedule\nThe next input variable we'll create is a proxy for a team's strength of schedule. Not all schedules are created equal. Some teams will play only a handful of tournament quality teams throughout the regular season, while other teams could see a large percentage of tournament caliber teams. Without addressing the difficulty of different schedules, it's nearly impossible to reconcile the quality of all other stats.  "},{"metadata":{"trusted":true},"cell_type":"code","source":"# We'll calculate a team's average margin of victory for each season. \nreg_season['mov_diff_avg'] = reg_season.groupby(['Season','Team'])['mov'].transform('mean')\n\n# This step creates the opponent's average margin of victory. \nreg_season['mov_opp_avg'] = reg_season.groupby(['Season','Opponent'])['mov'].transform('mean')\n\n# Finally, we take the average of a team's opponent's average margin of victory. \n# When a team's opponents have a strong average of an average margin of victory, it means a team has played a difficult schedule.\nreg_season['schd_strngth'] = reg_season.groupby(['Season','Team'])['mov_opp_avg'].transform('mean')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The next step is similar to the strength of schedule calculation. This time, we want to see how a team's offense performs in relation to the offenses of its opponents. It's a little tricky to explain but an example makes it simpler. If a team's average offense for a season was 75 points and its opponents allowed an average of 72 points, this team performed 3 points better than its opponent's opponents. This metric helps to control for a weaker strength of schedule."},{"metadata":{"trusted":true},"cell_type":"code","source":"# We calculate a team's opponent's average score allowed. \nreg_season['avg_def_opp'] = reg_season.groupby(['Season','Opponent'])['Team_Score'].transform('mean') \n\n# We take the average of that number. \nreg_season['avg_off_adj'] = reg_season.groupby(['Season','Team'])['avg_def_opp'].transform('mean') \n\n# Finally we take the difference of our average offensive performance versus our opponents average defensive performance. \nreg_season['off_adj'] = reg_season['avg_off'] - reg_season['avg_off_adj'] ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This step does a few things:\nIt brings in a few other inputs we'll need to derive additional metrics and it brings in a team's conference affiliation. After extensive testing, knowing if a team is in a mid-major conference, a 'minor' conference, the Pac-12 or SEC proved to be informative to the models. While there is no official designation for what constitutes a mid-major or a minor conference, by using data going back to 2003, it's fairly clear by looking at conference winning percentages and bids in the NCAA tournament what group a team should be in. Major conferences were assigned their own variable because they received the most at-large bids by a large margin. Mid-majors have won between 34%-52% of their NCAA tournament games having between 27-89 bids. Minors have won between 0%-24% and the big six conferences are all greater than 56% with all having over 150 bids. "},{"metadata":{"trusted":true},"cell_type":"code","source":"# These two lines help us to determine if a team won its last game of the season. It's a proxy for winning a conference championship.   \nreg_season['max_gameday'] = reg_season.groupby(['Season','Team'])['DayNum'].transform('max')\nreg_season['final_game'] = np.where(reg_season['max_gameday'] == reg_season['DayNum'],1,0)\n\n# We join in the conference dataset. The conferences and conference groupings are assigned a binary variable. The Pac-10 and Pac-12 are treated as the same conference. \nreg_season = pd.merge(reg_season, conference, left_on = ['Season','Team'],  right_on=['Season','Team'])\nreg_season['SEC'] = np.where(reg_season['ConfAbbrev'].isin(['sec']),1,0)\nreg_season['P12'] = np.where(reg_season['ConfAbbrev'].isin(['pac_ten','pac_twelve']),1,0)\nreg_season['Minors'] = np.where(reg_season['ConfAbbrev'].isin(['wac','ivy','mac','sun_belt','big_west','ovc','southern','a_sun','aec','maac'\n                                                               ,'patriot','big_sky','southland','big_south','meac','nec','summit','swac','mid_cont']),1,0)\nreg_season['Mid_majors'] = np.where(reg_season['ConfAbbrev'].isin(['a_ten','cusa','mwc','mvc','wcc','horizon','caa','aac']),1,0) \nreg_season = reg_season[(reg_season.Season>=2003)] \n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<div id=\"intLink3\"></div>\n# Modified Pythagorean Expectation\n\nFirst, we'll group the dataset using only the variables that will be in the final models. Because of the way the data was originally structured, we can take the mean of all calculated columns. After the data is grouped, we will calculate a modified version of the Pythagorean Expectation which is explained [here](https://en.wikipedia.org/wiki/Pythagorean_expectation#Use_in_basketball). Different exponents have been used but after using an optimization process, 9.5 for offense and 9.2 for defensive were chosen as they were the most predictive for NCAA Tournament modeling. Generally, the exponent is the same in the numerator and the denominator but not in our case. "},{"metadata":{"trusted":true},"cell_type":"code","source":"# Group the dataset by team and season. This gives us one record per team per year which is the needed structure for the final model.    \nseason_grp = reg_season.groupby(['Season','Team'])['mov_diff_avg','avg_off','avg_def','wp','off_adj','mov_min','schd_strngth','SEC','P12','Minors','Mid_majors'].mean().reset_index(drop=False)\n\n# We calculate a modified version of Pythagorean Expectation. These results will overestimate a team's quality of play because of the differences in exponents but that's fine. \nseason_grp['pythag'] = (season_grp['avg_off']**9.5)/((season_grp['avg_off']**9.5)+(season_grp['avg_def']**9.2))\n\n# We'll take the difference between a team's modified Pythagorean Expectation and actual win percentage. \nseason_grp['pythag_overage'] = season_grp['pythag'] - season_grp['wp']\nseason_grp = season_grp[(season_grp.Season>=2003)] \n\n# Referencing back to the reg_season data frame. This code creates a data frame that tells us if a team won its final game.    \nwon_final = reg_season[(reg_season.final_game ==1)]\nwon_final['won_final_game'] = np.where(won_final['mov'] > 0,1,0)\nwon_final = won_final.iloc[:,[0,2,25]]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<div id=\"intLink4\"></div>\n# Massey Ordinals\nNow, we'll move on to the Massey Ordinals data frame. As mentioned above, I'm only using data from 2003 and later with the reason being that's as far back as the Massey Ordinals go. I only use systems that go back to 2003 and have existed every year since. We'll drop the systems where media and coaches vote on the quality of teams: the USA and AP. \n<p>Besides using the ordinals, we'll create four derived input variables. They are a ratio using the WLK system, and the max, min and standard deviation from the group of eight systems we use. The ratio looks at a team's ordinal on either day 99 or 100 (depending on the year) and their ordinal on the last day of the season (133 except 2020 which is 128). Other approaches like the difference of ratings over time didn't prove valuable. A ratio > 1 means a team has a lower (better) ordinal than it did approximately five weeks prior suggesting it's trending in the right direction. "},{"metadata":{"trusted":true},"cell_type":"code","source":"# Due to some team/mapping issues, there were some teams duplicated and other missing for 2020. An update was made available so we'll filter out 2020 in the \n# original ordinals dataset and then concatenate the updated file as a fix.\nmassey_ord = massey_ord[(massey_ord.Season <= 2019 )] \nmassey_ord_2020  \nframes = [massey_ord,massey_ord_2020]\nmassey_ord = pd.concat(frames)\n\n# We'll create two datasets to create the ratio. Any system could have been used but the WLK system had no data integrity issues for the timeframe around days 99/100. \n# Other systems could be worth looking into.  \n  \nmassey_99 = massey_ord[((massey_ord.RankingDayNum == 99) | (massey_ord.RankingDayNum == 100)) & (massey_ord.SystemName == 'WLK' )]\nmassey_133 = massey_ord[(massey_ord.RankingDayNum == 133) & (massey_ord.SystemName == 'WLK' ) \n                        |((massey_ord.RankingDayNum == 128) & (massey_ord.SystemName == 'WLK' ) & (massey_ord.Season == 2020)) ]\nmassey_ratio = pd.merge(massey_99,massey_133,how='inner', on =['Season','TeamID'])\nmassey_ratio = massey_ratio.iloc[:,[0,3,4,7]]\nmassey_ratio['trend_ratio'] = massey_ratio['OrdinalRank_x'] / massey_ratio['OrdinalRank_y']\nmassey_ratio = massey_ratio.iloc[:,[0,1,4]]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The second part of the Massey Ordinals file code ultimately pivots the data from a long to wide format. Additionally, we need to identify all systems that have existed for the last 18 years and exist this year. The models performed better with more seasons. "},{"metadata":{"trusted":true},"cell_type":"code","source":"# This is a test to only include systems where all seasons exist and existed in 2020.\nmassey_ord = massey_ord[(massey_ord.Season>=2003)] \nm_test = massey_ord.groupby(['Season', 'SystemName'])['RankingDayNum'].max().reset_index(drop=False)\nm_test['seasons_count'] = m_test.groupby(['SystemName'])['RankingDayNum'].transform('count')\n\n# Once we have the systems, we store them in a list and use that list to filter our dataset down to only those results. \n# We remove the voting systems (USA Today and AP polls) because they only have ratings for a small set of teams. \n# In the future, I will look for a way to use them because they \n# are aligned with the perception of a team's quality more so than its actual strength.\n\nsystems_list = m_test[(m_test.seasons_count==18) & (m_test.Season==2020)]\nordinal_name = list(systems_list['SystemName'])\nordinal_name.remove('USA') \nordinal_name.remove('AP') \nmassey_ord = massey_ord[massey_ord.SystemName.isin(ordinal_name)]\n\n# We then find the last ranking day of the season and filter to that day. The data is pivoted to make it more usable for analysis. \nmassey_ord['max_day_num'] = massey_ord.groupby(['Season', 'SystemName', 'TeamID'])['RankingDayNum'].transform('max')\nmassey_ord = massey_ord[(massey_ord.RankingDayNum == massey_ord.max_day_num )]\nmassey_ord = massey_ord.iloc[:,[0,3,4,2]]\nmassey_ord = pd.pivot_table(massey_ord, index = ['Season','TeamID'], columns = 'SystemName', values = 'OrdinalRank')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The code below finds the min, max, and standard deviation for each system. Other metrics were tried but didn't prove to be useful. Having the actual system ratings and not just the ordinals would provide additional insight. There is significant difference in the quality of the best team and 20th as compared to the 180th and 200th. "},{"metadata":{"trusted":true},"cell_type":"code","source":"# The code below finds the min, max and standard deviation across the row for each team/season for all systems. \nmassey_ord['min_ord'] = massey_ord.min(axis=1)\nmassey_ord['max_ord'] = massey_ord.max(axis=1)\nmassey_ord['std_dev'] = np.std(massey_ord.iloc[ :,0:9],axis=1)\nmassey_ord = massey_ord.reset_index(drop=False)\n\n# We join these results with the ratio results from above. Finally, renaming the column makes joining cleaner later. \nmassey_ord = pd.merge(massey_ord,massey_ratio,how='inner',on = ['Season','TeamID'])\nmassey_ord.rename(columns={'TeamID':'Team'}, inplace = True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<div id=\"intLink5\"></div>\n# Aggregate the Data\nBelow we join our datasets into a training dataset to model the at-large teams. The imported file 'at_large' is a data frame of all teams, seasons, and booleans for at-large selections and conference winners. The results were scraped from Wikipedia. If a team won a conference and received an automatic berth, it will be filtered from the training dataset. "},{"metadata":{"trusted":true},"cell_type":"code","source":"# The four datasets are joined. We drop two columns that were needed in early calculations but won't be used for either problem. \ntrain = pd.merge(conference, season_grp, how='left', left_on = ['Season','Team'], right_on = ['Season','Team'])\ntrain = pd.merge(train, won_final, how='left', left_on = ['Season','Team'], right_on = ['Season','Team'])\ntrain = pd.merge(train,     massey_ord, how='left', left_on = ['Season','Team'], right_on = ['Season','Team'])\ntrain = train[(train.Season >= 2003)]\ntrain.drop(labels=['avg_off','avg_def'], inplace=True, axis=1)\n\n# After extensive testing, these columns returned the best results. \n# We only need seasons 2003-19. It worked out that losing some games due to the cancellation of some conferences tournament games should\n# have minimal impact on our model. None of the input variables were directly dependent on conference tournament results. This is an avenue I didn't look into but I suspect \n# knowing a team’s results in its conference tournament would have been predictive.\nat_large_train = train[['Season','Team','ConfAbbrev','wp','off_adj','pythag','mov_diff_avg','COL','MOR','RTH','SAG','WLK',\n                        'max_ord','schd_strngth','mov_min','SEC','P12','Mid_majors','Minors']]\n\n# This dataset contains our target variable. It also allows us to filter out the teams already guaranteed a place in the tournament due to winning their conference tournament. \nat_large = at_large.iloc[:,[0,1,2,3]]\nat_large['Team'] = at_large['Team'].astype(int) \n\n# We use an outer join. Any team that didn't get an invitation to the NCAA tournament will get an nan for the target column 'at_large'. We'll replace the nan's with zeros. \nat_large_train = pd.merge(at_large, at_large_train, how='outer', left_on = ['Season','Team'], right_on = ['Season','Team'])\nat_large_train['at_large'] = np.nan_to_num(at_large_train.at_large)\n\n# We filter out teams receiving an automatic bid and then drop that column. \nat_large_train = at_large_train[(at_large_train.auto_bid != 1)]\nat_large_train.drop(labels=['auto_bid'], inplace=True, axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# The second training dataset to predict the seeding in the NCAA tournament is created similarly to the previous approach. Not all input variables are the same but since we created everything\n# needed earlier, after we join all of the files, we'll select the predictive columns by name\n\nseeding_train = pd.merge(seeds_df, season_grp, how='left', left_on = ['Season','Team'], right_on = ['Season','Team'])\nseeding_train = pd.merge(seeding_train,     won_final, how='left', left_on = ['Season','Team'], right_on = ['Season','Team'])\nseeding_train = pd.merge(seeding_train,     massey_ord, how='left',   left_on = ['Season','Team'], right_on = ['Season','Team'])\nseeding_train = seeding_train[(seeding_train.Season >= 2003)]\nseeding_train.drop(labels=['avg_off','avg_def'], inplace=True, axis=1)\n\nseeding_train = seeding_train[['Season','Team','seed_int','wp','pythag_overage','mov_diff_avg','COL','DOL','MOR','POM','RTH','SAG',\n                        'WLK','WOL','min_ord','max_ord','std_dev','trend_ratio','schd_strngth','Minors','won_final_game']]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<div id=\"intLink6\"></div>\n# Model Creation: At-Large Selections\n\nWe're finally at the point where we get to see the results for the first model. Using data from 2003-2016, we create a dataset that we'll use for cross validation. We'll leave the years 2017-19 for validation. The goal is to have the validation data perform as well as or better than the cross validation results. The reason may not be intuitive but makes for sound logic. We're modeling this as a binary decision: will a team receive an at-large bid? The model can only interpret the input data and provide a probability of a selection. It doesn't reconcile for the limited number of available bids directly. It should approximate historical results; however, it will find too many deserving teams in some years and too few in others. For the validation results, we'll take the top 36 selections by probability - independent of whether or not it's above 0.5.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"cv_df = at_large_train[(at_large_train['Season'] < 2017)]\ntarget = cv_df[(cv_df['Season'] < 2017)]['at_large']\ncv_df = cv_df.iloc[:,4:]\nlgb_df = lgb.Dataset(cv_df, label=target)\n\n# The results for the cross fold validation have an error rate of 1.87%. Speaking in general terms, given that there are ~ 350 college basketball teams \n# and 32 conference champions receiving automatic bids, we are selecting 36 teams out of 320. If the model doesn't select any teams, \n# it will have an error rate of ~ 11% as it will be right in 284 out of 320 instances. \n\nparam = {'num_leaves': [2], \n         'objective': 'binary',\n          'metric':['binary_error'],\n          'learning_rate': [85/1000], \n          'min_data': [10],              \n          'max_depth' : [-1],\n          'min_hessian':7/10,    \n          'colsample_bytree':37/100,\n          'colsample_bynode':92/100,   \n          'lambda_l2': (1/20),    \n          'lambda_l1':(1/100), \n          'max_bin':84,                 \n          'bagging_fraction':(6/100),       \n          'bagging_freq':6,   \n          }\n\nclf_mod = lgb.cv(params = param, \n          nfold = 7, \n          train_set  = lgb_df, \n          num_boost_round = 1000,      \n          verbose_eval = 10,         \n          early_stopping_rounds = 10)\n\n# The best iteration is num_boost_round which we'll use in during the model creation. That number is 41.\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The code below creates a model using data from 2003-2016. We'll predict on seasons 2017-2019. While a low overall error rate is encouraging, we \nreally care about the true positives - how many teams does it correctly choose out of the 36 available slots per year for the three years of our validation data. \nThe printed output will give us the overall accuracy we can compare to our cross validation results. It will also return the number of correct \nselections for each of the three validations seasons. \n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Basic data preparation to create the model \nat_large_df = at_large_train[(at_large_train['Season'] < 2017)]\nat_large_target = at_large_df[(at_large_df['Season'] < 2017)]['at_large']\nat_large_df = at_large_df.iloc[:,4:]\nat_large_pred = at_large_train[(at_large_train['Season'] >= 2017) & (at_large_train['Season'] <= 2020)]   \nat_large_pred_join = at_large_pred.iloc[:,[0,1,2,3]].reset_index(drop=True)  \nat_large_pred = at_large_pred.iloc[:,4:]\nlgb_df = lgb.Dataset(at_large_df, label=at_large_target)  \n\n# We create the model, predict on seasons 2017-19 and join the results back to a data frame with team and season information.  \nclf_mod = lgb.train(param, lgb_df, num_boost_round = 41)\nmodel_preds = pd.DataFrame(clf_mod.predict(at_large_pred)) #lgbm\npred_df = pd.merge(at_large_pred_join,model_preds,left_index=True, right_index=True)              \npred_df.rename(columns={0:'pred'}, inplace= True)\n\n# To see the results per year, we need to rank the probabilities by year. \n# By creating a 'pred_rank' column looking at only the top 36 selection and a 'correct' column looking at all selections,\n# it makes it simpler to return results. Consider that while this is a classification problem returning probabilities, \n# the cutoff is dynamic for every season. \npred_df['pred_rank'] = pred_df.groupby(['Season'])['pred'].rank(ascending=False)\npred_df['rank_pred'] = np.where(((pred_df['pred_rank'] <= 36) & (pred_df['at_large']==1)) ,1,0)  \npred_df['correct'] = np.where(((pred_df['pred_rank'] > 36 ) & (pred_df['at_large']== 0))\n                              |((pred_df['pred_rank'] <= 36) & (pred_df['at_large']== 1)) ,1,0)   \navg_accuracy = pred_df[(pred_df.Season<2020)]['correct'].mean()\ncorrect_count = pred_df['rank_pred'].sum()                \nby_season_splits = pred_df.groupby(['Season'])['rank_pred'].sum().reset_index(drop=False) \n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<div id=\"intLink7\"></div>\n# At-Large Selection Model Results\nThe results are good. The overall error rate is slightly lower than our cross validation results at 1.3% and we correctly predicted the 2017 season. 2018 and 2019 gave us 33 of 36 correct selections. For the three combined years, we correctly picked 102 of 108 at-large bids. The actual process is done via a 10 member committee with members serving a five year term with two members being replaced every year. To be able to replicate their selections this closely given the turnover is exciting."},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Accuracy between 2017-2019: ' + str(round(avg_accuracy,3)))\nprint('Total Correct Selections: ' + str(round(correct_count,4)) + ' out of 108')\nprint(by_season_splits[(by_season_splits.Season <=2019)])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<div id=\"intLink8\"></div>\n# At-Large Selection Feature Importance\nInterestingly, the plot results don't show that all of the input variables have feature importance. I suspect the primary reason it doesn't is because I originally modeled seasons 2003-2014 and left 2015-2019 for validation. Nonetheless, the results are no worse. Seeing that five of the inputs are computer models suggests that the committee uses them for decision support. This makes sense when trying to objectively evaluate between hundreds of teams even if not all teams are being seriously considered. "},{"metadata":{"trusted":true},"cell_type":"code","source":"\n\nfeature_imp = pd.DataFrame(sorted(zip(clf_mod.feature_importance(),at_large_df.columns)), columns=['Value','Feature'])\nfeature_imp = feature_imp[(feature_imp.Value>0)].sort_values('Value', ascending = False)\n\nplt.figure(figsize=(10, 5))\n#f, ax = plt.subplots(figsize=(6, 15))\nsns.set(style='darkgrid')\nax = sns.barplot(x='Feature', y='Value', data=feature_imp, palette = 'summer') #     color='teal')\nplt.title('LightGBM Variable Importance',fontsize=18)\nplt.ylabel('LGB Value')\nplt.xlabel('Input Variables',fontsize=14) \nplt.show()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<div id=\"intLink9\"></div>\n# Wrangling the 2020 data\nEarlier, we created the predictions for this year for at-large teams in the pred_df data frame. We’ll filter it for 2020 and use it to assist us in determining the conference tournament ‘winners’. But before we start, there is an issue to resolve with our process. Only 13 of the 32 conference tournaments crowned a champion meaning there are still 19 at-large positions for 19 conferences that remain unfilled. We could create a model or many models to determine the winners of those tournaments, however for simplicity’s sake, we’ll use the at-large model’s probabilities instead as it’s a reasonable proxy for team strength. The teams with the highest probability of being selected for an at-large bid will be given their conference’s automatic bid. Some of the conference tournaments were partially complete at the time they were cancelled, so any team that was eliminated won’t be considered. There are two conferences where all teams share the exact same probability of ~ .005%: the MEAC and SWAC. In these two instances we’ll take the top remaining seed from each which is NC Central and Prairie View. In four of the power conferences - ACC, Big East, Big 12 and the SEC - the top two or three performing teams share the same probability of 96.96% so we'll use the same approach and award the highest seed the conference automatic bid. The remaining teams will still receive an at-large bid. We then need to update the team selection for the American East Conference from 1467 to 1271 as team 1467 was eliminated in their conference tournament and 1271 was the highest remaining seed. Lastly, we have to update the variable ‘won_final_game’ for the 32 teams receiving auto-bids and for the 36 at-large teams as it was created to identify conference tournament winners."},{"metadata":{"trusted":true},"cell_type":"code","source":"# We'll use our prediction data frame and filter it on only 2020. We'll then create two lists, one for all conferences and one for conferences with tournament champions. \npred_df_2020 = pred_df[(pred_df.Season == 2020)]\nconf_list = pred_df_2020['ConfAbbrev'].tolist()\n# This step eliminates duplicates\nconf_list = list(dict.fromkeys(conf_list))\nauto_bid_list = at_large_2020['conf'].tolist()\n# We remove everything from list one that is in list two and return a list we can filter on below. \nconf_list = [x for x in conf_list if x not in auto_bid_list]\n\n# We loop through our conferences and find the team with the highest probability to create a data frame. There can be duplicated records for some conference due to ties in the probabilities. \nconf_auto_bid = pd.DataFrame()\nfor i in conf_list:\n    conf_df = pred_df_2020[(pred_df_2020.ConfAbbrev == i)]\n    conf_df = conf_df.iloc[:,:6].reset_index(drop=True)\n    conf_df['max_conf'] = conf_df.groupby('ConfAbbrev')['pred'].transform(max)\n    conf_df = conf_df[(conf_df.pred == conf_df.max_conf)]\n    conf_auto_bid = conf_auto_bid.append(conf_df)    \n\n# We need to split our data frame into two. Any conference without duplication will be given the automatic bid. \nconf_auto_bid['conf_count'] = conf_auto_bid.groupby('ConfAbbrev')['pred'].transform('count')\nconf_auto_bid_a = conf_auto_bid[(conf_auto_bid.conf_count == 1)]\n\n# We filter on conferences with multiple top teams. I've previously verified the top seeds to the teams in the top_seed_filter list.\n# The list will then be used to select the top conference tournament seeded teams. \nconf_auto_bid_b = conf_auto_bid[(conf_auto_bid.conf_count > 1)]\ntop_seed_filter = [1341,1300,1199,1166,1242,1246]\nconf_auto_bid_b = conf_auto_bid_b[conf_auto_bid_b['Team'].isin(top_seed_filter)]\n\n# We'll combine our results into our auto-bid data frame and update the AEC conference results as mentioned above as team 1467 \n# had the highest probability but lost. We then assign all teams in this data frame a zero as an 'at-large' selection and a one for an 'auto_bid'\n# Lastly, we need to change column locations to match the structure of the at_large_2020 below.\nconf_auto_bid_f = conf_auto_bid_a.append(conf_auto_bid_b)\nconf_auto_bid_f['Team'] = conf_auto_bid_f['Team'].replace(1467,1436)\nconf_auto_bid_f['at_large'] = 0\nconf_auto_bid_f = conf_auto_bid_f.iloc[:,:4]\nconf_auto_bid_f['auto_bid'] = 1\nconf_auto_bid_f =  conf_auto_bid_f.iloc[:,[0,1,2,4,3]]\n\n# This code appends the 13 conference tournament winners to our 19 modeled 'conference champions'. \nat_large_2020.rename(columns={'conf':'ConfAbbrev'}, inplace = True)\nat_large_2020 = at_large_2020.append(conf_auto_bid_f)\n\n\n# Now that we know who our 32 at-large teams are, we can use our at-large model predictions to select the 38 at-large teams for 2020!\n# We create a list of the automatic bids and filter the teams in it. \nauto_bid_list = at_large_2020['Team'].tolist()\nat_large_2020_pred = pred_df_2020[~pred_df_2020['Team'].isin(auto_bid_list)]\n\n# Now that we have only teams available for an at-large selection, we rank them by the model prediction and take the top 36 teams. \n# We'll print rankings 36-37 to verify there is no tie at the 36th position. There isn't so no tie breaks are needed and we can take the top 36 teams.\nat_large_2020_pred['pred_rank'] = at_large_2020_pred.groupby(['Season'])['pred'].rank(ascending=False)\nprint('The 36th and 37th teams have different probabilites for an at-large selection')\nprint(at_large_2020_pred[(at_large_2020_pred.pred_rank <= 37) & (at_large_2020_pred.pred_rank >= 36)].sort_values('pred',ascending = False).iloc[:,[0,1,3,4,5]])\nprint()\n# We'll print teams 33-36 to see who were the last four in as us basketball geeks love that kind of thing.\n# In order, they are: Texas Tech, Providence, Cinci, Miss St. \nprint('Last four in')\nprint(at_large_2020_pred[(at_large_2020_pred.pred_rank <= 36) & (at_large_2020_pred.pred_rank >= 33)].sort_values('pred',ascending = False).iloc[:,[0,1,3,4,5]])\nprint()\n# We'll also print teams 37-40 to see who was on the 'bubble' and barely missed a selection. \n# They are: Arizona St, NC State, Arkansas, Memphis - all from different conferences. \n# We'll see two teams with probabilities  > 50% left out. This is a top heavy year. \nprint('First four out')\nprint(at_large_2020_pred[(at_large_2020_pred.pred_rank <= 40) & (at_large_2020_pred.pred_rank >= 37)].sort_values('pred',ascending = False).iloc[:,[0,1,3,4,5]])\nprint()\n\nat_large_2020_pred = at_large_2020_pred[(at_large_2020_pred.pred_rank <= 36)]\nat_large_2020_pred['at_large'] = 1\nat_large_2020_pred['auto_bid'] = 0\nat_large_2020_pred.drop(labels = ['pred','pred_rank','rank_pred','correct'], inplace=True, axis=1)\nat_large_2020_pred = at_large_2020_pred.iloc[:,[3,0,1,2,4]]\nNCAA_2020_tourn_teams = at_large_2020_pred.append(at_large_2020)\n\n#For some peace of mind, we'll print the number of unique conferences and total teams. The outputs are exactly as we'd hoped.  \nprint(NCAA_2020_tourn_teams.groupby('Season')['ConfAbbrev'].nunique())\nprint()\nprint(NCAA_2020_tourn_teams.groupby('Season')['Team'].count())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<div id=\"intLink10\"></div>\n# Model Creation: Team Seeding\nThe next step is to assign each team a seed. This is a more difficult modeling problem because each team can be assigned one of 16 seeds. We won't see accuracy near the performance of the At-Large Selection model.  \nThe goal is to correctly predict a team's seeding. As a secondary goal, we'll also measure the accuracy of a result of +/- 1 of the seed. \nAfter extensive testing, the best results came from modeling this as a regression problem, ranking the results, and assigning those rankings a seed.\nUsing a multi-classification approach returned accuracy results in the mid 40%'s. We'll see from the validation set that we can do better than that. "},{"metadata":{"trusted":true},"cell_type":"code","source":"\ntrain_split = seeding_train[(seeding_train['Season'] < 2017)]\ntarget = train_split[(train_split['Season'] < 2017)]['seed_int']\ntrain_split = train_split.iloc[:,3:]                       \nlgb_df = lgb.Dataset(train_split, label=target)\n\nparam = {'num_leaves': [2], \n         'objective': 'mae',\n         'learning_rate': [43/1000], \n         'min_data': [17],                \n          'max_depth' : [-1],\n          'min_hessian':0/10,    \n          'colsample_bytree':53/100,\n          'colsample_bynode':25/100,     \n          'lambda_l2': (192/20),    \n          'lambda_l1':(0/100), \n          'max_bin':163,                      \n          'bagging_fraction':(33/100),        \n          'bagging_freq':36,       \n          'verbose':1}\n\nclf_mod_2 = lgb.cv(params = param, \n          nfold = 7, \n          train_set  = lgb_df, \n          num_boost_round = 3000,\n          verbose_eval = 100,\n          early_stopping_rounds = 100)\n\n# The best iteration is 514 which is what we'll use to train the model.   \n# The cross validation error is < 1. We'll see how this translates in the validation set. ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# We performed cross validation above and now we'll compare those results with the unseen data from years 2017-2019. If our results don't \n# resemble the results above, we know we've overfit. It's a little abstract to know if we actually have overfit due to the complexity of \n# the problem, so we'll rely on the knowledge that modeling as a multiclass problem cross validated in the mid 40%. \n# We split the data out by the 2017 season \ntrain_split = seeding_train[(seeding_train['Season'] < 2017)]\ntarget = train_split[(train_split['Season'] < 2017)]['seed_int']\ntrain_split = train_split.iloc[:,3:]\npred_split = seeding_train[(seeding_train['Season'] >= 2017)]\npred_split_join = pred_split.iloc[:,[0,1,2]].reset_index(drop=True)\npred_split = pred_split.iloc[:,3:]            \n\n# We create the lightgbm dataset, model and predict the results.   \nlgb_df = lgb.Dataset(train_split, label=target)\nclf_mod_2 = lgb.train(param, lgb_df, num_boost_round = 514)\npred_df = pd.DataFrame(clf_mod_2.predict(pred_split))  \n\n# We build a usable data frame \npred_df = pd.merge(pred_split_join,pred_df,left_index=True, right_index=True) \npred_df['pred'] = pred_df.idxmax(axis=1)\npred_df.rename(columns={0:'pred_regression'}, inplace= True)\n\n# We rank our predictions by season. This gives us rankings between 1-68. Simply dividing by 4 doesn't address the issue of needing six 11 seeds and six 16 seeds.\n# It would instead return four 17 seeds. If the rank is >= 45, we subtract 2 from the rank and do it again if it's over 65. By doing this, when we divide the rank by 4, \n# we'll get six 11 and 16 seeds matching what has historically happened over the last three seasons. \npred_df['pred_rank'] = pred_df.groupby('Season')['pred_regression'].rank()\npred_df['pred_rank'] = np.where (pred_df['pred_rank'] >=  45, pred_df['pred_rank'] -2, pred_df['pred_rank'])\npred_df['pred_rank'] = np.where (pred_df['pred_rank'] >= 65, pred_df['pred_rank'] -2, pred_df['pred_rank'])        \npred_df['pred_rank'] = np.ceil((pred_df['pred_rank']/4))                   \npred_df = pred_df.iloc[:,[0,1,2,5]]\n\n# We'll create variables to measure performance for the validation set and break it out by year. \n# The stated goal is to predict the correct seed. Closer is better especially if this approach is viewed as a guide for human decision support. \n# We'll create a metric and print it showing predictions within one seed.\n\npred_df['correct_seeds'] = np.where(pred_df.seed_int == pred_df.pred_rank,1,0) \npred_df['seed_diff'] = abs(pred_df['pred_rank']-pred_df['seed_int'])\npred_df['within_one'] = np.where(pred_df.seed_diff <= 1,1,0) \naccuracy = pred_df['correct_seeds'].mean()\nwithin_one = pred_df['within_one'].mean()\nby_season_splits = pred_df.groupby(['Season'])['correct_seeds'].mean().reset_index(drop=False) \n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<div id=\"intLink11\"></div>\n# Seeding results\nWe were able to predict > 52% which was better than the multi-class approaches I tied. \nWe were also able to predict within one seed 87.75% of the time. I didn't expect a result this high. It's interesting. "},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Total average accuracy between 2017-2019: ' + str(round(accuracy,3)))\nprint('Prediction Percent within one seed: ' + str(round(within_one,4)))\nprint(by_season_splits)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<div id=\"intLink12\"></div>\n# Team Seeding Model Feature Importance\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# We'll print the feature importance for the model which uses nearly all inputs. Massey Ordinals represent six of the eight most important variables with min_ord being a derivative. \n# For both models, RTH is the most important feature. Win percentage is the only non-ordinal in both models - winning matters. \n\nfeature_imp = pd.DataFrame(sorted(zip(clf_mod_2.feature_importance(),train_split.columns)), columns=['Value','Feature'])\nfeature_imp = feature_imp[(feature_imp.Value>0)].sort_values('Value', ascending = False)\n\nplt.figure(figsize=(25, 5))\nsns.set(style='darkgrid')\nax = sns.barplot(x='Feature', y='Value', data=feature_imp, palette = 'summer') #     color='teal')\nplt.title('LightGBM Variable Importance',fontsize=18)\nplt.ylabel('LGB Value')\nplt.xlabel('Input Variables',fontsize=14) \nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n# Seeds to Input Variable Correlation \n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"\n# This code creates a correlation plot between all variables. \ncorr_2017 = seeding_train[(seeding_train['Season'] < 2017)].iloc[:,2:].corr()\nnames = ['Seed','WP%','PY_Overage','MOV_avg','COL','DOL','MOR','POM','RTH','SAG','WLK','WOL','Min_Ord','Max_Ord','STD_DEV','Ord_Trnd','SOS','Minor','Won_Last']\nfig = plt.figure(figsize=(25, 25))\nax = fig.add_subplot(111)\ncax = ax.matshow(corr_2017, vmin=-1, vmax=1,  cmap = 'plasma', interpolation = 'nearest',filternorm = False ) \nfig.colorbar(cax)\nticks = np.arange(0,19,1)\nax.set_xticks(ticks)\nax.set_yticks(ticks)\nax.set_xticklabels(names)\nax.set_yticklabels(names)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<div id=\"intLink13\"></div>\n# The Tournament We Wanted:\nWe’ve created two models: the first determines the at-large selections and the second model's seedings. Up to this point, these two processes haven’t been linked. We’ve predicted the first step but didn’t use the results in the second step. The reason is because we didn't achieve 100% accuracy and the second model won't be accurate without correct target variables. However, we didn’t get the at-large selections for 2020, so for this year, the results from the at-large picks will be imported into the seeding model. The results will give us an idea of what may have happened this year had the season finished. "},{"metadata":{"trusted":true},"cell_type":"code","source":"# Data wrangling to calculate the 2020 results. This process is similar to the one above for the validation dataset. \n# For any questions about the code, reference the above. \nseeding_pred = pd.merge(NCAA_2020_tourn_teams, season_grp, how='left', left_on = ['Season','Team'], right_on = ['Season','Team'])\nseeding_pred = pd.merge(seeding_pred,     won_final, how='left', left_on = ['Season','Team'], right_on = ['Season','Team'])\nseeding_pred = pd.merge(seeding_pred,     massey_ord, how='left',   left_on = ['Season','Team'], right_on = ['Season','Team'])\nseeding_pred = seeding_pred[(seeding_pred.Season == 2020)]\nseeding_pred.drop(labels=['avg_off','avg_def'], inplace=True, axis=1)\n\nseeding_pred = seeding_pred[['Season','Team','wp','pythag_overage','mov_diff_avg','COL','DOL','MOR','POM','RTH','SAG',\n                        'WLK','WOL','min_ord','max_ord','std_dev','trend_ratio','schd_strngth','Minors','auto_bid']]\nseeding_pred.rename(columns={'auto_bid':'won_final_game'}, inplace = True)\n\nseeding_pred_a = seeding_pred.iloc[:,2:]\nseeding_pred_join = seeding_pred.iloc[:,:2] \npred_df_2020 = pd.DataFrame(clf_mod_2.predict(seeding_pred_a))\npred_df_2020 =  pd.merge(seeding_pred_join,pred_df_2020, left_index = True, right_index = True)\n\npred_df_2020.rename(columns={0:'pred_regression'}, inplace= True)\npred_df_2020['pred_rank'] = pred_df_2020['pred_regression'].rank()\npred_df_2020['pred_rank'] = np.where (pred_df_2020['pred_rank'] >=  45, pred_df_2020['pred_rank'] -2, pred_df_2020['pred_rank'])\npred_df_2020['pred_rank'] = np.where (pred_df_2020['pred_rank'] >= 65, pred_df_2020['pred_rank'] -2, pred_df_2020['pred_rank'])        \npred_df_2020['seed'] = np.ceil((pred_df_2020['pred_rank']/4))   ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<div id=\"intLink14\"></div>\n# The 2020 final results\nThere was a fair amount of work to get to this point. We print the results out at a seed level for all to see along with a chart. How do we compare to other predictions?\nAfter looking at a few of the larger sporting news outlets, it's really interesting to compare my result with those from [Andy Katz of ESPN fame](https://www.ncaa.com/news/basketball-men/article/2020-03-09/ncaa-predictions-andy-katzs-projections-2020-tournament), [Bleacher Report](http://bleacherreport.com/articles/2880016-ncaa-bracketology-2020-real-time-seed-and-region-projections-for-all-68-teams), and [SB Nation](http://www.sbnation.com/college-basketball/2020/3/16/21181438/ncaa-bracket-predictions-2020-march-madness-mens-tournament-dayton-gonzaga-national-champion). Andy Katz posted his on March 11th and the other two had an update after the tournament cancellation. I didn't do an extensive comparison but here are a few interesting call-outs.\n* Duke is a three or four seed in AK, BR, and SBN but I have them as a one seed. \n* I have BYU higher than the other three predictions as a four seed.\n* We all agree that Creighton and Florida State are two seeds and that last year's Champion Virginia is a six seed. \n* I have Arizona St as the first team out (even with a 58% at-large chance) but the other predictions have them between an 8 and 10 seed.\n\nUltimately, our results aren't that different from the national publications which is useful as a sanity check. Over the last three years, our model correctly seeded an average of 34 of 68 seeds and 59.7 of 68 within one seed (either +/-). It would have been interesting to have an actual comparison to the actual results. Here's to 2021! "},{"metadata":{"trusted":true},"cell_type":"code","source":"# We'll bring in the conferences and team names and print the results by seed. \n# Worth noting, over the last three years there have been six 11 seeds and six 16 seeds so that's what you'll see here. \n\npred_df_2020 = pd.merge(pred_df_2020, conference, how='left', left_on = ['Season','Team'], right_on = ['Season','Team'])\npred_df_2020 = pd.merge(pred_df_2020, teams, how='left', left_on = ['Team'], right_on = ['Team'])\npred_df_2020 = pred_df_2020.iloc[:,[0,6,5,4,2]].sort_values('pred_regression')\n\n# Raw data for those that prefer it not in chart form. \nfor i in range(0,17,1):\n    print(pred_df_2020[(pred_df_2020.seed == i)].iloc[:,:4])\n    print()     ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Plotting he seeding for all 68 teams\nf, ax = plt.subplots(figsize=(15,25))\nplt.title('Team''s Predicted Seeding for 2020' ,fontsize=20)\nsns.barplot(x=\"seed\", y=\"TeamName\", data=pred_df_2020,\n            label=\"Total\", palette =\"plasma\")\nax.set( ylabel=\"Team Names\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<div id=\"intLink15\"></div>\n# 2020 Results by Conference\nThe bar charts below show the results broken out by conference for 2020. The first represents number of bids with the second representing average seeding. \nAny league not shown had one bid with the lowest seed being a 12. This suggests that having multiple good teams in a conference helps with the seeding as well. "},{"metadata":{"trusted":true},"cell_type":"code","source":"# We create a grouped data frame to chart our results for both bid count and average seeding\ntest_df = pred_df_2020.groupby('ConfAbbrev')['seed'].agg(['count','mean']).sort_values(by = 'count', ascending = False).reset_index(drop=False)\n\nplt.figure(figsize=(20, 5))\nsns.set(style='darkgrid')\nax = sns.barplot(x='ConfAbbrev', y='count', data=test_df.head(10), palette='autumn')\nplt.title('Bids per conference',fontsize=20)\nplt.ylabel('Selections')\nplt.xlabel('Conference',fontsize=14)\nplt.show()\n\nplt.figure(figsize=(20, 5))\nsns.set(style='darkgrid')\nax = sns.barplot(x='ConfAbbrev', y='mean', data=test_df.head(10).sort_values(by = 'mean', ascending = True).reset_index(drop=False), palette='autumn')\nplt.title('Average seed per conference',fontsize=20)\nplt.ylabel('avg seed')\nplt.xlabel('Conference',fontsize=14)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<div id=\"intLink16\"></div>\n# Observations for years 2017-2019\nNow that we've seen what 2020 would have looked like, we're going to look back at 2017-2019. Does our model agree with the selections? Are some seeds easier to model than others? Is there bias to the major conferences? \nThe charts show a few interesting points according to my model:\n* Accurately predicting the middle seeds is more difficult than the lower and higher seeds. As a fan, it makes sense because the quality of teams in the middle is much more difficult to determine and seemingly more subjective than either end of the seeding spectrum. \n* Some conferences are much easier to predict than others. This is partly a function of the number of teams selected from a conference. The difference in model accuracy between the SEC and Pac-12 is fairly large. \n* Of the bigger conferences, the SEC and ACC seem to - on average - get a slightly favorable seeding with the Big Ten getting unfavorable seedings. \n* In only two instances over three years did my model disagree with a seeding by three or more positions - once to the benefit of South Carolina and once to the detriment of Wichita St. Only one team made the list twice - Michigan St - both over and under seeded. \n* Overall, as much as fans love to debate fairness of athletics, it seems there's no significant bias for the NCAA men's tournament. If there were major discrepancies, it would mean the seeding process or my modeling process is lacking which is something I wouldn't want to see. "},{"metadata":{"trusted":true},"cell_type":"code","source":"# We'll do some grouping of data to create bar charts. \npred_df = pd.merge(pred_df, conference, how='left', left_on = ['Season','Team'], right_on = ['Season','Team'])\npred_df = pd.merge(pred_df, teams, how='left', left_on = ['Team'], right_on = ['Team'])\npred_df = pred_df.iloc[:,[0,8,7,3,2,5,4,6]].sort_values('Season')\n# Seeding accuracy \npred_df_seed = pred_df.groupby('seed_int')['correct_seeds','within_one','seed_diff'].mean().reset_index(drop = False) #.sort_values(by = 'mean', ascending = False).reset_index(drop=False)\n# Conference level accuracy and under and overseeding\npred_df['overseeded'] = pred_df['seed_int'] - pred_df['pred_rank'] \npred_df_conf_mean = pred_df.groupby('ConfAbbrev')['correct_seeds','within_one','seed_diff','overseeded'].mean().reset_index(drop = False)\npred_df_conf_count = pred_df.groupby('ConfAbbrev')['correct_seeds'].count().reset_index(drop = False)\npred_df_conf_mean_overseed = pred_df_conf_mean[(pred_df_conf_mean.overseeded !=0)]\n# Team level under and over seeding\npred_df_filter = pred_df[(pred_df.seed_diff > 1)]\npred_df_filter['overseeded'] =  pred_df_filter['seed_int'] - pred_df_filter['pred_rank'] \npred_df_filter = pred_df_filter.iloc[:,[0,1,2,3,4,8]].sort_values(by = 'overseeded', ascending = False).reset_index(drop=True)    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Plotting the outputs of our model's accuracy for each seed.\nplt.figure(figsize=(30, 5))\nsns.set(style='darkgrid')\nax = sns.barplot(x='seed_int', y='correct_seeds', data=pred_df_seed.head(16), palette='cool')\nplt.title('Average Accuracy per Seed: 2017-2019',fontsize=18)\nplt.ylabel('Accuracy')\nplt.xlabel('Seeding',fontsize=14) \nplt.show()\n\n# Plotting the outputs of our model's accuracy for each conference.\nplt.figure(figsize=(30, 5))\nsns.set(style='darkgrid')\nax = sns.barplot(x='ConfAbbrev', y='correct_seeds', data=pred_df_conf_mean.sort_values('correct_seeds'), palette='cool')\nplt.title('Average Accuracy per Conference: 2017-2019',fontsize=18)\nplt.ylabel('Accuracy')\nplt.xlabel('Seeding',fontsize=14) \nplt.show()\n\n# Plotting the outputs of our model's prediction differnce between avg conference seeding \nplt.figure(figsize=(30, 5))\nsns.set(style='darkgrid')\nax = sns.barplot(x='ConfAbbrev', y='overseeded', data=pred_df_conf_mean_overseed.sort_values('overseeded', ascending = False), palette = 'plasma') #     color='teal')\nplt.title('Average Underseeding per Conference: 2017-2019',fontsize=18)\nplt.ylabel('Underseeding')\nplt.xlabel('Conference',fontsize=14) \nplt.show()\n\n# Plotting the outputs of our model's prediction differnce at a team level\nplt.figure(figsize=(30, 5))\nsns.set(style='darkgrid')\nax = sns.barplot(x='TeamName', y='overseeded', data=pred_df_filter, palette = 'plasma') #     color='teal')\nplt.title('Underseeding: Teams 2017-2019',fontsize=18)\nplt.ylabel('Underseeding')\nplt.xlabel('Team',fontsize=14) \nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<div id=\"intLink17\"></div>\n# A Few Final Words:\nWe created a model to determine the at-large teams that would have been selected and fed the results to a second model to determine the seedings for 2020. \nWe looked at the model's performance compared to what actually happened for the last three years. \nWe transformed raw data into something usable. It was a journey.  \nIf you made it this far, I appreciate your time and hope you found something of interest within this notebook. Sports analytics is a passion and hobby of mine. As much as the cancellation of this year's tournament was a disappointment, being able to model the at-large and seeding process was rewarding and a nice way for me personally to offset not being able to model the tournament itself. \n\nKaggle folks: Thanks for making this competition available for the first time and here's to an NCAA ML Comp in 2021!\n\nFinally, this is my first notebook as a contributor. Feedback is welcome and greatly appreciated. \n\nBen\n\n"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}