{"cells":[{"metadata":{},"cell_type":"markdown","source":"I have been a member of Kaggle for over 2 years now, but my offline job with Lagos State University as a day job and other periods distributed with updating Handsonlabs Software Academy research & development projects leaves me with very narrow time to contribute to kernels and notebooks on Kaggle. I would strongly suggest, with all due respect to the Adminstrators of Kaggle to try and develop preferably an Android Mobile App to keep updates and contributions handy.\nHowever, I hope to create more time soon to be more active in competitions, discussions etc. \n\nIn order to tell NCCA Google Cloud & NCAA® March Madness Analytics ability of a team to “stay in the game” as well as increase their chance to win late in the contest, I take specific analytics from historical point of view as well as common exploratory processes of the data before actually doing the predictictive algorithm analytics."},{"metadata":{},"cell_type":"markdown","source":"Firstly, let's import the basic modules:\n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport csv\nimport math\nimport os\nimport pandas as pd\nimport random\nimport sklearn\nfrom sklearn.model_selection import KFold\nfrom sklearn import linear_model\n#from sklearn import cross_validation, linear_model\nfrom sklearn.model_selection import cross_validate\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.model_selection import cross_val_score\nfrom sklearn.model_selection import train_test_split\nimport datetime, time\nimport sys\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport re\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.\ndf_fdr =\"/kaggle/input/march-madness-analytics-2020/2020DataFiles/2020DataFiles/2020-Mens-Data/MDataFiles_Stage1/\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Credits to Original code by @spencerthayer (2017) open source published on Github. This is an improved version.**** The First Part tackkles the men Stage1 & 2 data. While the second part of this algorithm handles the Women Stage1 and Stage2 data respectively."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"if len(sys.argv) == 1:\n    print(\"- - - - - - - - - - - - - - - - - - - - - - - - -\")\n    print(\"NO DEFINED YEAR! QUITTING!\")\n    print(str(time.time()))\n    print(\"Use 'python3 NCAA.py [YEAR]' to define the year.\")\n    print(\"- - - - - - - - - - - - - - - - - - - - - - - - -\")\n    quit()\nelse:\n    theYear = sys.argv[1]\n\nprint(\"Generating results for \" + theYear +\".\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2. Seasons Data Analysis\n\nSpecific year that the tournament has been dand event. Of cos the most recent being 2020."},{"metadata":{"trusted":true},"cell_type":"code","source":"seasondata=pd.read_csv(df_fdr +'MSeasons.csv')\nseasondata.tail()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Total held seasons including the current\nseasondata['Season'].count()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"mteams=pd.read_csv(df_fdr +'MTeams.csv')\nmteams.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"** This part is for identifying the seeds for all teams in each NCAA® tournament, for all seasons of historical data. Geeting to know the seeds for all teams in the NCAA tournament from historic perspective**"},{"metadata":{"trusted":true},"cell_type":"code","source":"mseeds=pd.read_csv(df_fdr +'MNCAATourneySeeds.csv')\nmseeds.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"mseeds = pd.merge(mseeds, mteams,on='TeamID')\nmseeds.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Spliting regions from the Seeds\n\nmseeds['Region'] = mseeds['Seed'].apply(lambda x: x[0][:1])\nmseeds['Seed'] = mseeds['Seed'].apply(lambda x: int(x[1:3]))\nprint(mseeds.head())\nprint(mseeds.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Teams with maximum top seeds\nfig = plt.gcf()\nfig.set_size_inches(10, 6)\ncolors = ['dodgerblue', 'plum', '#F0A30A','#8c564b','orange','green','yellow'] \n\nmseeds[mseeds['Seed'] ==1]['TeamName'].value_counts()[:10].plot(kind='bar',color=colors,linewidth=2,edgecolor='black')\nplt.xlabel('Number of times in Top seeded positions')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"mcompactre = pd.read_csv(df_fdr +'MRegularSeasonCompactResults.csv')\nmcompactre.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#  Score Avergae for Win vs Loss over a certain past\nx = mcompactre.groupby('Season')[['WScore','LScore']].mean()\n\nfig = plt.gcf()\nfig.set_size_inches(28, 12)\nplt.plot(x.index,x['WScore'],marker='o', markerfacecolor='green', markersize=12, color='green', linewidth=4)\nplt.plot(x.index,x['LScore'],marker=7, markerfacecolor='red', markersize=12, color='red', linewidth=4)\nplt.legend()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#df_fdr = '/kaggle/working/'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"files = os.listdir(df_fdr)\nnames = [x.split('.')[0] for x in files]\npath_list = [f'{df_fdr}{x}' for x in files]\ndf_dict = {}\nfor num, name in enumerate(names):\n    try:\n        df_dict[name] = pd.read_csv(path_list[num])\n    except:\n        print(f'{name} did not load')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Summary of files:\\n___________________________')\nfor name, df in df_dict.items():\n    print(f'{name}: {df.shape}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## > DATA\npath_datasets = df_fdr\n\ndf_rs_c_res = pd.read_csv(path_datasets + 'MRegularSeasonCompactResults.csv')\ndf_rs_d_res = pd.read_csv(path_datasets + 'MRegularSeasonDetailedResults.csv')\ndf_teams = pd.read_csv(path_datasets + 'MTeams.csv')\ndf_seeds = pd.read_csv(path_datasets + 'MNCAATourneySeeds.csv')\ncoaches = pd.read_csv(path_datasets + 'MTeams.csv')\ndf_tourn = pd.read_csv(path_datasets + 'MNCAATourneyCompactResults.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## > DATA CLEANING\n# clean team information\n\ndf_teams_cl = df_teams.iloc[:,:2]\n\n## > DATA CLEANING\n# clean seed information\n\ndf_seeds_cl = df_seeds.loc[:, ['TeamID', 'Season', 'Seed']]\n\ndef clean_seed(seed):\n    s_int = int(seed[1:3])\n    return s_int\n\ndef extract_seed_region(seed):\n    s_reg = seed[0:1]\n    return s_reg\n\ndf_seeds_cl['seed_int'] = df_seeds_cl['Seed'].apply(lambda x: clean_seed(x))\ndf_seeds_cl['seed_region'] = df_seeds_cl['Seed'].apply(lambda x: extract_seed_region(x))\ndf_seeds_cl['top_seeded_teams'] = np.where(df_seeds_cl['Seed'].isnull(), 0, 1)\n\ndf_seeds_cl.drop(labels=['Seed'], inplace=True, axis=1) # This is the string label\ndf_seeds_cl.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## > DATA CLEANING\n# create games dataframe WINNERS\n\ndef new_name_w_1(old_name):\n    match = re.match(r'^L', old_name)\n    if match:\n        out = re.sub('^L','', old_name)\n        return out + '_opp'\n    return old_name\n\ndef new_name_w_2(old_name):\n    match = re.match(r'^W', old_name)\n    if match:\n        out = re.sub('^W','', old_name)\n        return out\n    return old_name\n\ndef prepare_stats_extended_winners(df_in, df_seed_in, df_teams_in):\n    df_in['poss'] = df_in['WFGA'] + 0.475*df_in['WFTA'] - df_in['WOR'] + df_in['WTO']\n    df_in['opp_poss'] = df_in['LFGA'] + 0.475*df_in['LFTA'] - df_in['LOR'] + df_in['LTO']\n    df_in['off_rating'] = 100*(df_in['WScore'] / df_in['poss'])\n    df_in['def_rating'] = 100*(df_in['LScore'] / df_in['opp_poss'])\n    df_in['net_rating'] = df_in['off_rating'] - df_in['def_rating']\n    df_in['pace'] = 48*((df_in['poss']+df_in['opp_poss'])/(2*(240/5)))\n    \n    df_in = df_in.rename(columns={'WTeamID':'TeamID', \n                                  'WLoc':'_Loc',\n                                  'LTeamID':'TeamID_opp',\n                                  'WScore':'Score_left', \n                                  'LScore':'Score_right'})\n    \n    df_seeds_opp = df_seed_in.rename(columns={'TeamID':'TeamID_opp',\n                                              'seed_int':'seed_int_opp',\n                                              'seed_region':'seed_region_opp',\n                                              'top_seeded_teams':'top_seeded_teams_opp'})\n    \n    df_out = pd.merge(left=df_in, right=df_seeds_cl, how='left', on=['Season', 'TeamID'])\n    df_out = pd.merge(left=df_out, right=df_seeds_opp, how='left', on=['Season', 'TeamID_opp'])\n    df_out = pd.merge(left=df_out, right=df_teams_in, how='left', on=['TeamID'])\n    \n    df_out['DayNum'] = pd.to_numeric(df_out['DayNum'])\n    df_out['win_dummy'] = 1\n    \n    df_out['seed_int'] = np.where(df_out['seed_int'].isnull(), 20, df_out['seed_int'])\n    df_out['seed_region'] = np.where(df_out['seed_region'].isnull(), 'NoTour', df_out['seed_region'])\n    df_out['top_seeded_teams'] = np.where(df_out['top_seeded_teams'].isnull(), 0, df_out['top_seeded_teams'])\n    \n    df_out['seed_int_opp'] = np.where(df_out['seed_int_opp'].isnull(), 20, df_out['seed_int_opp'])\n    df_out['seed_region_opp'] = np.where(df_out['seed_region_opp'].isnull(), 'NoTour', df_out['seed_region_opp'])\n    df_out['top_seeded_teams_opp'] = np.where(df_out['top_seeded_teams_opp'].isnull(), 0, df_out['top_seeded_teams_opp'])\n    \n    df_out = df_out.rename(columns=new_name_w_1)\n    df_out = df_out.rename(columns=new_name_w_2)\n    \n    return df_out\n\ndf_games_w = prepare_stats_extended_winners(df_rs_d_res, df_seeds_cl, df_teams_cl)\n\ndf_games_w.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## > DATA CLEANING\n# create games dataframe LOSERS\n\ndef new_name_l_1(old_name):\n    match = re.match(r'^W', old_name)\n    if match:\n        out = re.sub('^W','', old_name)\n        return out + '_opp'\n    return old_name\n\ndef new_name_l_2(old_name):\n    match = re.match(r'^L', old_name)\n    if match:\n        out = re.sub('^L','', old_name)\n        return out\n    return old_name\n\ndef prepare_stats_extended_losers(df_in, df_seed_in, df_teams_in):\n    df_in['poss'] = df_in['LFGA'] + (0.475*df_in['LFTA']) - df_in['LOR'] + df_in['LTO']\n    df_in['opp_poss'] = df_in['WFGA'] + (0.475*df_in['WFTA']) - df_in['WOR'] + df_in['WTO']\n    df_in['off_rating'] = 100*(df_in['LScore'] / df_in['poss'])\n    df_in['def_rating'] = 100*(df_in['WScore'] / df_in['opp_poss'])\n    df_in['net_rating'] = df_in['off_rating'] - df_in['def_rating']\n    df_in['pace'] = 48*((df_in['poss']+df_in['opp_poss'])/(2*(240/5)))\n    \n    df_in = df_in.rename(columns={'LTeamID':'TeamID', \n                                  'LLoc':'_Loc',\n                                  'WTeamID':'TeamID_opp',\n                                  'LScore':'Score_left', \n                                  'WScore':'Score_right'})\n    \n    df_seeds_opp = df_seed_in.rename(columns={'TeamID':'TeamID_opp',\n                                              'seed_int':'seed_int_opp',\n                                              'seed_region':'seed_region_opp',\n                                              'top_seeded_teams':'top_seeded_teams_opp'})\n    \n    df_out = pd.merge(left=df_in, right=df_seeds_cl, how='left', on=['Season', 'TeamID'])\n    df_out = pd.merge(left=df_out, right=df_seeds_opp, how='left', on=['Season', 'TeamID_opp'])\n    df_out = pd.merge(left=df_out, right=df_teams_in, how='left', on=['TeamID'])\n    \n    df_out['DayNum'] = pd.to_numeric(df_out['DayNum'])\n    df_out['win_dummy'] = 0\n    \n    df_out['seed_int'] = np.where(df_out['seed_int'].isnull(), 20, df_out['seed_int'])\n    df_out['seed_region'] = np.where(df_out['seed_region'].isnull(), 'NoTour', df_out['seed_region'])\n    df_out['top_seeded_teams'] = np.where(df_out['top_seeded_teams'].isnull(), 0, df_out['top_seeded_teams'])\n    \n    df_out['seed_int_opp'] = np.where(df_out['seed_int_opp'].isnull(), 20, df_out['seed_int_opp'])\n    df_out['seed_region_opp'] = np.where(df_out['seed_region_opp'].isnull(), 'NoTour', df_out['seed_region_opp'])\n    df_out['top_seeded_teams_opp'] = np.where(df_out['top_seeded_teams_opp'].isnull(), 0, df_out['top_seeded_teams_opp'])\n\n    df_out = df_out.rename(columns=new_name_l_1)\n    df_out = df_out.rename(columns=new_name_l_2)\n    \n    return df_out\n\ndf_games_l = prepare_stats_extended_losers(df_rs_d_res, df_seeds_cl, df_teams_cl)\n\ndf_games_l.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## > MERGE\n\ndf_games_t = pd.concat([df_games_w,df_games_l], sort=True)\n\n## > AGGREGATED STATS BY TEAM AND SEASON\n\ndef aggr_stats(df):\n    d = {}\n    d['G'] = df['win_dummy'].count()\n    d['W'] = df['win_dummy'].sum()\n    d['L'] = np.sum(df['win_dummy'] == 0)\n    d['G_vs_topseeds'] = np.sum(df['top_seeded_teams_opp'] == 1)\n    d['W_vs_topseeds'] = np.sum((df['win_dummy'] == 1) & (df['top_seeded_teams_opp'] == 1))\n    d['L_vs_topseeds'] = np.sum((df['win_dummy'] == 0) & (df['top_seeded_teams_opp'] == 1))\n    d['G_last30D'] = np.sum((df['DayNum'] > 100))\n    d['W_last30D'] = np.sum((df['win_dummy'] == 1) & (df['DayNum'] > 100))\n    d['L_last30D'] = np.sum((df['win_dummy'] == 0) & (df['DayNum'] > 100))\n    d['G_H'] = np.sum((df['_Loc'] == 'H'))\n    d['W_H'] = np.sum((df['win_dummy'] == 1) & (df['_Loc'] == 'H'))\n    d['L_H'] = np.sum((df['win_dummy'] == 0) & (df['_Loc'] == 'H'))\n    d['G_A'] = np.sum((df['_Loc'] == 'A'))\n    d['W_A'] = np.sum((df['win_dummy'] == 1) & (df['_Loc'] == 'A'))\n    d['L_A'] = np.sum((df['win_dummy'] == 0) & (df['_Loc'] == 'A'))\n    d['G_N'] = np.sum((df['_Loc'] == 'N'))\n    d['W_N'] = np.sum((df['win_dummy'] == 1) & (df['_Loc'] == 'N'))\n    d['L_N'] = np.sum((df['win_dummy'] == 0) & (df['_Loc'] == 'N'))\n    \n    d['PS'] = np.mean(df['Score_left'])\n    d['PS_H'] = np.mean(df['Score_left'][df['_Loc'] == 'H'])\n    d['PS_A'] = np.mean(df['Score_left'][df['_Loc'] == 'A'])\n    d['PS_N'] = np.mean(df['Score_left'][df['_Loc'] == 'N'])\n    d['PS_last30D'] = np.mean(df['Score_left'][df['DayNum'] > 100])\n    \n    d['PA'] = np.mean(df['Score_right'])\n    d['PA_H'] = np.mean(df['Score_right'][df['_Loc'] == 'H'])\n    d['PA_A'] = np.mean(df['Score_right'][df['_Loc'] == 'A'])\n    d['PA_N'] = np.mean(df['Score_right'][df['_Loc'] == 'N'])\n    d['PA_last30D'] = np.mean(df['Score_right'][df['DayNum'] > 100])\n    \n    d['poss_m'] = np.mean(df['poss'])\n    d['opp_poss_m'] = np.mean(df['opp_poss'])\n    d['off_rating_m'] = np.mean(df['off_rating'])\n    d['def_rating_m'] = np.mean(df['def_rating'])\n    d['net_rating_m'] = np.mean(df['net_rating'])\n    d['pace_m'] = np.mean(df['pace'])\n    \n    d['off_rating_m_last30D'] = np.mean(df['off_rating'][df['DayNum'] > 100])\n    d['def_rating_m_last30D'] = np.mean(df['def_rating'][df['DayNum'] > 100])\n    d['net_rating_m_last30D'] = np.mean(df['net_rating'][df['DayNum'] > 100])\n    \n    d['off_rating_m_vs_topseeds'] = np.mean(df['off_rating'][df['top_seeded_teams_opp'] == 1])\n    d['def_rating_m_vs_topseeds'] = np.mean(df['def_rating'][df['top_seeded_teams_opp'] == 1])\n    d['net_rating_m_vs_topseeds'] = np.mean(df['net_rating'][df['top_seeded_teams_opp'] == 1])\n    \n    return pd.Series(d)\n\n\ndf_agg_stats = df_games_t.\\\n                          groupby([df_games_t['Season'], \n                                   df_games_t['TeamID'],\n                                   df_games_t['TeamName'],\n                                   df_games_t['seed_int'],\n                                   df_games_t['seed_region']], \n                                  as_index=False).\\\n                          apply(aggr_stats).\\\n                          reset_index()\n\n\ndf_agg_stats['w_pct'] = df_agg_stats['W'] / df_agg_stats['G']\ndf_agg_stats['w_pct_last30D'] = df_agg_stats['W_last30D'] / df_agg_stats['G_last30D']\ndf_agg_stats['w_pct_vs_topseeds'] = df_agg_stats['W_vs_topseeds'] / df_agg_stats['G_vs_topseeds']\n\ndf_agg_stats.head(20)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## > DATA CLEANING \n\n# prepare tournament dataset\ndef prepare_tournament_datasets(df_tourn_in, df_agg_stats_in):\n    \n    df_tourn_in['TeamID'] = df_tourn_in[['WTeamID','LTeamID']].min(axis=1)\n    df_tourn_in['TeamID_opp'] = df_tourn_in[['WTeamID','LTeamID']].max(axis=1)\n    df_tourn_in['win_dummy'] = np.where(df_tourn_in['TeamID'] == df_tourn_in['WTeamID'], 1, 0)\n    df_tourn_in['delta'] = np.where(df_tourn_in['win_dummy'] == 1,\n                                    df_tourn_in['WScore'] - df_tourn['LScore'],\n                                    df_tourn_in['LScore'] - df_tourn['WScore'])\n    df_tourn_in['Score_left'] = np.where(df_tourn_in['win_dummy'] == 1,\n                                         df_tourn_in['WScore'],\n                                         df_tourn_in['LScore'])\n    df_tourn_in['Score_right'] = np.where(df_tourn_in['win_dummy'] == 1,\n                                          df_tourn_in['LScore'],\n                                          df_tourn_in['WScore'])\n                                 \n    df_teams_gr_left = df_agg_stats_in.loc[:,['Season', 'TeamID',\n                                              'w_pct', 'seed_int', \n                                              'net_rating_m_last30D',\n                                              'net_rating_m_vs_topseeds',\n                                              'net_rating_m']].\\\n                  rename(columns={'w_pct':'w_pct_left',\n                                  'seed_int':'seed_int_left', \n                                  'net_rating_m_last30D':'net_rating_m_last30D_left', \n                                  'net_rating_m_vs_topseeds':'net_rating_m_vs_topseeds_left', \n                                  'net_rating_m':'net_rating_m_left'})\n    \n    df_teams_gr_right = df_agg_stats_in.loc[:,['Season', 'TeamID',\n                                               'w_pct', 'seed_int',\n                                               'net_rating_m_last30D',\n                                               'net_rating_m_vs_topseeds',\n                                               'net_rating_m']].\\\n                  rename(columns={'TeamID':'TeamID_opp',\n                                  'w_pct':'w_pct_right',\n                                  'seed_int':'seed_int_right', \n                                  'net_rating_m_last30D':'net_rating_m_last30D_right', \n                                  'net_rating_m_vs_topseeds':'net_rating_m_vs_topseeds_right', \n                                  'net_rating_m':'net_rating_m_right'})\n    \n    df_tourn_out = pd.merge(left=df_tourn_in, \n                            right=df_teams_gr_left, \n                            how='left', on=['Season', 'TeamID'])\n    df_tourn_out = pd.merge(left=df_tourn_out, \n                            right=df_teams_gr_right, \n                            how='left', on=['Season', 'TeamID_opp'])\n\n    df_tourn_out['delta_w_pct'] = df_tourn_out['w_pct_left'] - \\\n                                         df_tourn_out['w_pct_right']\n\n\n    df_tourn_out['delta_seed_int'] = df_tourn_out['seed_int_left'] - \\\n                                          df_tourn_out['seed_int_right']\n\n\n    df_tourn_out['delta_net_rating_m'] = df_tourn_out['net_rating_m_left'] - df_tourn_out['net_rating_m_right']\n    \n    df_tourn_out['delta_net_rating_m_last30D'] = df_tourn_out['net_rating_m_last30D_left'] - df_tourn_out['net_rating_m_last30D_right']\n    \n    df_tourn_out['delta_net_rating_m_vs_topseeds'] = df_tourn_out['net_rating_m_vs_topseeds_left'] - df_tourn_out['net_rating_m_vs_topseeds_right']\n    \n    df_out = df_tourn_out.loc[:, ['Season', 'DayNum',\n                                  'TeamID', 'TeamID_opp',\n                                  'Score_left', 'Score_right',\n                                  'win_dummy', \n                                  'delta', 'NumOT', 'delta_w_pct', \n                                  'delta_net_rating_m_last30D',\n                                  'delta_net_rating_m_vs_topseeds',\n                                  'delta_net_rating_m', 'delta_seed_int']]\n                                    \n    return df_out\n\n                                    \ndf_tourn_cl = prepare_tournament_datasets(df_tourn, df_agg_stats)                                    \ndf_tourn_cl[(df_tourn_cl['Season'].isin([2015, 2016, 2017, 2018]))].head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## > DUKE RS\ndf_agg_stats[(df_agg_stats['TeamName'] == 'Duke') & (df_agg_stats['Season'] == 2019)].head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## > DUKE TOURNAMENT\ndf_tourn_cl[((df_tourn_cl['TeamID'] == 1181) | (df_tourn_cl['TeamID_opp'] == 1181)) & \\\n            (df_tourn_cl['Season'] == 2019)].head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## > DATA VIZ RS\nsns.set(style=\"ticks\", color_codes=True)\n\ndf_teams_gr = df_agg_stats.loc[:,['w_pct',\n                                  'net_rating_m', 'net_rating_m_last30D', \n                                  'net_rating_m_vs_topseeds', 'pace_m']]\n\ndf_teams_gr = df_teams_gr.fillna(0)\n\n#df_teams_gr.describe()\nsns.pairplot(df_teams_gr, palette=\"Set1\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## > DATA VIZ TOURNEY\nsns.set(style=\"ticks\", color_codes=True)\n\ndf_tourn_cl_gr = df_tourn_cl[(df_tourn_cl['Season'].isin([2015, 2016, 2017, 2018]))].reindex()\n\ndf_tourn_cl_gr = df_tourn_cl_gr.loc[:,['win_dummy',\n                                       'delta_net_rating_m_last30D',\n                                       'delta_net_rating_m_vs_topseeds',\n                                       'delta_net_rating_m',  \n                                       'delta_seed_int']]\n\nfig, ax = plt.subplots(figsize=(11, 7))\nsns.boxplot(x=\"variable\", y=\"value\", hue = 'win_dummy', ax=ax, \n            data=pd.melt(df_tourn_cl_gr, id_vars='win_dummy'), palette=\"Set2\")\nplt.xticks(rotation=45)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## > DATA VIZ TOURNEY\ndf_tourn_cl_gr = df_tourn_cl[(df_tourn_cl['Season'].isin([2015, 2016, 2017, 2018]))].reindex()\n\ndf_tourn_cl_gr = df_tourn_cl_gr.loc[:,['win_dummy',\n                                       'delta_w_pct']]\n\nfig, ax = plt.subplots(figsize=(9, 7))\nsns.boxplot(x=\"variable\", y=\"value\", hue = 'win_dummy', ax=ax, \n            data=pd.melt(df_tourn_cl_gr, id_vars='win_dummy'), palette=\"Set2\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## > Correlation\n# Compute the correlation matrix\ndf_tourn_cl_gr = df_tourn_cl[(df_tourn_cl['Season'].isin([2015, 2016, 2017, 2018]))].reindex()\n\ndf_tourn_cl_gr = df_tourn_cl_gr.loc[:,['win_dummy',\n                                       'delta_net_rating_m_last30D',\n                                       'delta_net_rating_m_vs_topseeds',                                       \n                                       'delta_net_rating_m',  \n                                       'delta_w_pct',\n                                       'delta_seed_int']].fillna(0)\n\ncorr = df_tourn_cl_gr.corr()\nfig, ax = plt.subplots(figsize=(11, 7))\nsns.heatmap(corr, cmap=\"YlGnBu\", ax = ax)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## > AR\ndef somers2_py(x, y):\n    \n    from sklearn.metrics import roc_auc_score\n    \n    C = roc_auc_score(y, x)\n    Dxy = (2 * roc_auc_score(y, x))  - 1\n    \n    return Dxy, C\n\ndef apply_somers(df):\n    \n    d = {}\n    \n    dxy, cxy = somers2_py(df['value'],\n                          df['win_dummy'])\n    \n    d['Dxy'] = dxy\n    d['C'] = cxy\n    \n    \n    return pd.Series(d)\n\ndf_tourn_cl_gr = df_tourn_cl[(df_tourn_cl['Season'].isin([2015, 2016, 2017, 2018,2019]))].reindex()\n\ndf_tourn_cl_gr = df_tourn_cl_gr.loc[:,['win_dummy',\n                                       'delta_net_rating_m_last30D',\n                                       'delta_net_rating_m_vs_topseeds',                                       \n                                       'delta_net_rating_m',  \n                                       'delta_w_pct',\n                                       'delta_seed_int']].fillna(0)\n\ndf_ar = pd.melt(df_tourn_cl_gr, id_vars='win_dummy')\n\ndf_ar.groupby(['variable']).\\\n                          apply(apply_somers).\\\n                          reset_index().\\\n                          sort_values(by=['Dxy'], ascending=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Begin real Analysis"},{"metadata":{"trusted":true},"cell_type":"code","source":"timeString = str(time.time())\nbase_elo = 1600\nteam_elos = {}  # Reset each year.\nteam_stats = {}\nX = []\ny = []\nsubmission_data = []\nprediction_year = int(2019)\nfolder = df_fdr","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def calc_elo(win_team, lose_team, season):\n    winner_rank = get_elo(season, win_team)\n    loser_rank = get_elo(season, lose_team)\n\n    rank_diff = winner_rank - loser_rank\n    exp = (rank_diff * -1) / 400\n    odds = 1 / (1 + math.pow(10, exp))\n    if winner_rank < 2100:\n        k = 32\n    elif winner_rank >= 2100 and winner_rank < 2400:\n        k = 24\n    else:\n        k = 16\n    new_winner_rank = round(winner_rank + (k * (1 - odds)))\n    new_rank_diff = new_winner_rank - winner_rank\n    new_loser_rank = loser_rank - new_rank_diff\n\n    return new_winner_rank, new_loser_rank\n\n\ndef initialize_data():\n    for i in range(1985, int(2019)+1):\n        team_elos[i] = {}\n        team_stats[i] = {}\n\n\ndef get_elo(season, team):\n    try:\n        return team_elos[season][team]\n    except:\n        try:\n            # Get the previous season's ending value.\n            team_elos[season][team] = team_elos[season-1][team]\n            return team_elos[season][team]\n        except:\n            # Get the starter elo.\n            team_elos[season][team] = base_elo\n            return team_elos[season][team]\n\n\ndef predict_winner(team_1, team_2, model, season, stat_fields):\n    features = []\n\n    # Team 1\n    features.append(get_elo(season, team_1))\n    for stat in stat_fields:\n        features.append(get_stat(season, team_1, stat))\n\n    # Team 2\n    features.append(get_elo(season, team_2))\n    for stat in stat_fields:\n        features.append(get_stat(season, team_2, stat))\n\n    return model.predict_proba([features])\n\n\ndef update_stats(season, team, fields):\n    \"\"\"\n    This accepts some stats for a team and udpates the averages.\n\n    First, we check if the team is in the dict yet. If it's not, we add it.\n    Then, we try to check if the key has more than 5 values in it.\n        If it does, we remove the first one\n        Either way, we append the new one.\n    If we can't check, then it doesn't exist, so we just add this.\n\n    Later, we'll get the average of these items.\n    \"\"\"\n    if team not in team_stats[season]:\n        team_stats[season][team] = {}\n\n    for key, value in fields.items():\n        # Make sure we have the field.\n        if key not in team_stats[season][team]:\n            team_stats[season][team][key] = []\n        #Compare the last 10 games.\n        if len(team_stats[season][team][key]) >= 10:\n            team_stats[season][team][key].pop()\n        team_stats[season][team][key].append(value)\n\n\ndef get_stat(season, team, field):\n    try:\n        l = team_stats[season][team][field]\n        return sum(l) / float(len(l))\n    except:\n        return 0\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_team_dict():\n    team_ids = pd.read_csv(folder + 'MTeams.csv')\n    team_id_map = {}\n    for index, row in team_ids.iterrows():\n        team_id_map[row['TeamID']] = row['TeamName']\n    return team_id_map","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This part calls the build_season data function to handle the season data."},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_season_data(all_data):\n    # Calculate the elo for every game for every team, each season.\n    # Store the elo per season so we can retrieve their end elo\n    # later in order to predict the tournaments without having to\n    # inject the prediction into this loop.\n    print(\"Building season data.\")\n    for index, row in all_data.iterrows():\n        # Used to skip matchups where we don't have usable stats yet.\n        skip = 0\n\n        # Get starter or previous elos.\n        team_1_elo = get_elo(row['Season'], row['WTeamID'])\n        team_2_elo = get_elo(row['Season'], row['LTeamID'])\n\n        # Add 100 to the home team (# taken from Nate Silver analysis.)\n        if row['WLoc'] == 'H':\n            team_1_elo += 100\n        elif row['WLoc'] == 'A':\n            team_2_elo += 100\n\n        # We'll create some arrays to use later.\n        team_1_features = [team_1_elo]\n        team_2_features = [team_2_elo]\n\n        # print(\"Building arrays out of the stats.\")\n        # Build arrays out of the stats we're tracking..\n        for field in stat_fields:\n            team_1_stat = get_stat(row['Season'], row['WTeamID'], field)\n            team_2_stat = get_stat(row['Season'], row['LTeamID'], field)\n            if team_1_stat is not 0 and team_2_stat is not 0:\n                team_1_features.append(team_1_stat)\n                team_2_features.append(team_2_stat)\n            else:\n                skip = 1\n\n        if skip == 0:  # Make sure we have stats.\n            # Randomly select left and right and 0 or 1 so we can train\n            # for multiple classes.\n            if random.random() > 0.5:\n                X.append(team_1_features + team_2_features)\n                y.append(0)\n            else:\n                X.append(team_2_features + team_1_features)\n                y.append(1)\n\n        # AFTER we add the current stuff to the prediction, update for\n        # next time. Order here is key so we don't fit on data from the\n        # same game we're trying to predict.\n        if row['WFTA'] != 0 and row['LFTA'] != 0:\n            stat_1_fields = {\n                'score': row['WScore'],\n                'fgp': row['WFGM'] / row['WFGA'] * 100,\n                'fga': row['WFGA'],\n                'fga3': row['WFGA3'],\n                '3pp': row['WFGM3'] / row['WFGA3'] * 100,\n                'ftp': row['WFTM'] / row['WFTA'] * 100,\n                'or': row['WOR'],\n                'dr': row['WDR'],\n                'ast': row['WAst'],\n                'to': row['WTO'],\n                'stl': row['WStl'],\n                'blk': row['WBlk'],\n                'pf': row['WPF']\n            }\n            stat_2_fields = {\n                'score': row['LScore'],\n                'fgp': row['LFGM'] / row['LFGA'] * 100,\n                'fga': row['LFGA'],\n                'fga3': row['LFGA3'],\n                '3pp': row['LFGM3'] / row['LFGA3'] * 100,\n                'ftp': row['LFTM'] / row['LFTA'] * 100,\n                'or': row['LOR'],\n                'dr': row['LDR'],\n                'ast': row['LAst'],\n                'to': row['LTO'],\n                'stl': row['LStl'],\n                'blk': row['LBlk'],\n                'pf': row['LPF']\n            }\n            update_stats(row['Season'], row['WTeamID'], stat_1_fields)\n            update_stats(row['Season'], row['LTeamID'], stat_2_fields)\n        \n        # Now that we've added them, calc the new elo.\n        new_winner_rank, new_loser_rank = calc_elo(\n            row['WTeamID'], row['LTeamID'], row['Season'])\n        team_elos[row['Season']][row['WTeamID']] = new_winner_rank\n        team_elos[row['Season']][row['LTeamID']] = new_loser_rank\n\n        # Adding to make the processing look more dramatic.\n        print(new_winner_rank, stat_1_fields, team_1_features)\n        print(new_loser_rank, stat_2_fields, team_2_features)\n        print(\" \")\n\n    return X, y","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The Last part of this notebook employs certain functions to corroborate the analytics done so far and outputs the possible winners."},{"metadata":{"trusted":true},"cell_type":"code","source":"\nif __name__ == \"__main__\":\n    stat_fields = ['score', 'fga', 'fgp', 'fga3', '3pp', 'ftp', 'or', 'dr',\n                   'ast', 'to', 'stl', 'blk', 'pf']\n\n    initialize_data()\n    season_data = pd.read_csv(df_fdr + 'MRegularSeasonDetailedResults.csv')\n    tourney_data = pd.read_csv(df_fdr + 'MNCAATourneyDetailedResults.csv')\n    frames = [season_data, tourney_data]\n    all_data = pd.concat(frames)\n\n    # Build the working data.\n    X, y = build_season_data(all_data)\n\n    # Fit the model.\n    print(\"Fitting on %d samples.\" % len(X))\n\n    model = linear_model.LogisticRegression()\n\n    # Check accuracy.kfold_5 = KFold(n_splits = numFolds, shuffle=True)\n\n    print(\"Checking accuracy with cross-validation:\")\n    print(sklearn.model_selection.cross_val_score(\n        model, X, y, cv=10, scoring='accuracy', n_jobs=-1\n    ).mean())      \n    \n    model.fit(X, y)\n\n    # Now predict tournament matchups.\n    print(\"Getting teams.\")\n    seeds = pd.read_csv(df_fdr + 'MNCAATourneySeeds.csv')\n    # for i in range for year:\n    tourney_teams = []\n    for index, row in seeds.iterrows():\n        if row['Season'] == prediction_year:\n            tourney_teams.append(row['TeamID'])\n\n    # Build our prediction of every matchup.\n    print(\"Predicting matchups.\")\n    tourney_teams.sort()\n    for team_1 in tourney_teams:\n        for team_2 in tourney_teams:\n            if team_1 < team_2:\n                prediction = predict_winner(\n                    team_1, team_2, model, prediction_year, stat_fields)\n                label = str(prediction_year) + '_' + str(team_1) + '_' + \\\n                    str(team_2)\n                submission_data.append([label, prediction[0][0]])\n\n    # Write the results.\n    print(\"Writing %d results.\" % len(submission_data))\n    if not os.path.isdir(\"results\"):\n        os.mkdir(\"results\")\n    with open('/kaggle/working/results/submission.'+timeString+'.csv', 'w') as f:\n        writer = csv.writer(f)\n        writer.writerow(['id', 'pred'])\n        writer.writerows(submission_data)\n\n    # Now so that we can use this to fill out a bracket, create a readable version.\n    print(\"Outputting readable results.\")\n    team_id_map = build_team_dict()\n    readable = []\n    less_readable = []  # A version that's easy to look up.\n    for pred in submission_data:\n        parts = pred[0].split('_')\n        less_readable.append(\n            [team_id_map[int(parts[1])], team_id_map[int(parts[2])], pred[1]])\n        # Order them properly.\n        if pred[1] > 0.5:\n            winning = int(parts[1])\n            losing = int(parts[2])\n            proba = pred[1]\n        else:\n            winning = int(parts[2])\n            losing = int(parts[1])\n            proba = 1 - pred[1]\n        readable.append(\n            [\n                '%s beats %s: %f' %\n                (team_id_map[winning], team_id_map[losing], proba)\n            ]\n        )\n    with open('/kaggle/working/predictions.'+timeString+'.txt', 'w') as f:\n        writer = csv.writer(f)\n        writer.writerows(readable)\n    with open('/kaggle/working/predictions.'+timeString+'.csv', 'w') as f:\n        writer = csv.writer(f)\n        writer.writerows(less_readable)\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**References:**\n\n1. \tboyuan-li: Kaggle-GCP-NCAA-ML-2019 \n\thttps://github.com/boyuan-li/Kaggle-GCP-NCAA-ML-2019:\n\n2.        wdg3: Deep neural network that attempts to predict the NCAA \tMarch Madness tournament: https://github.com/wdg3/march-madness\n\n3. \tDecoding March Madness Kaggle Notebook:@Parul Pandey Data Science Evangelist at \tH2O.ai"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}