{"cells":[{"metadata":{},"cell_type":"markdown","source":"<div align='center'><font size=\"5\" color='#353B47'>A Notebook dedicated to Stacking/Ensemble methods</font></div>\n<div align='center'><font size=\"4\" color=\"#353B47\">Unity is strength</font></div>\n<br>\n<hr>","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"In this notebook, i'm going to cover various Prediction Averaging/Blending Techniques:\n\n1. Simple Averaging: Most participants are using just a simple mean of predictions generated by different models\n\n2. Rank Averaging: Use the \"rank\" of an input image instead it's prediction value. See public notebook Improve blending using Rankdata\n\n3. Weighted Averaging: Specify weights, say 0.5 each in case of two models WeightedAverage(p) = (wt1 x Pred1 + wt2 x Pred2 + … + wtn x Predn) where, n is the number of models, and sum of weights wt1+wt2+…+wtn = 1\n\n4. Stretch Averaging: Stretch predictions using min and max values first, before averaging Pred = (Pred - min(Pred)) / (max(Pred) - min(Pred))\n\n5. Power Averaging: Choose a power p = 2, 4, 8, 16 PowerAverage(p) = (Pred1^p + Pred2^p + … + Predn^p) / n Note: Power Averaging to be used only when all the models are highly correlated, otherwise your score may become worse.\n\n6. Power Averaging with weights: PowerAverageWithWeights(p) = (wt1 x Pred1^p + wt2 x Pred2^p + … + wtn x Predn^p)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# How to use\n\n* Create a dataset containing a folder with the models to be stacked\n* Add data in this notebook\n* Use the functions","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Import libraries","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nfrom scipy.stats import rankdata\nimport os\nimport re","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Stacking","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def Stacking(input_folder, \n             best_base, \n             output_path, \n             column_names, \n             cutoff_lo, \n             cutoff_hi):\n    \n    '''\n    To be tried on:\n        - a same model that is not deterministic (with randomness)\n        - a same model with folds (will need to define a meta model)\n    '''\n    \n    sub_base = pd.read_csv(best_base)\n    all_files = os.listdir(input_folder)\n    nb_files = len(all_files)\n    \n    # Test compliancy of arguments\n    assert type(input_folder) == str, \"Wrong type\"\n    assert type(best_base) == str, \"Wrong type\"\n    assert type(output_path) == str, \"Wrong type\"\n    assert type(cutoff_lo) in [float, int], \"Wrong type\"\n    assert type(cutoff_hi) in [float, int], \"Wrong type\"\n    assert (cutoff_lo >= 0) & (cutoff_lo <= 1) & (cutoff_hi >= 0) & (cutoff_hi <= 1), \"cutoff_lo and cutoff_hi must be between 0 and 1\"\n    assert len(column_names) == 2, \"Only two columns must be in column_names\"\n    assert type(column_names[0]) == str, \"Wrong type\"\n    assert type(column_names[1]) == str, \"Wrong type\"\n\n    \n    # Read and concatenate submissions\n    concat_sub = pd.DataFrame()\n    concat_sub[column_names[0]] = sub_base[column_names[0]]\n    for index, f in enumerate(all_files):\n        concat_sub[column_names[1]+str(index)] = pd.read_csv(input_folder + f)[column_names[1]]\n    print(\" ***** 1/4 Read and concatenate submissions SUCCESSFUL *****\")\n        \n    # Get the data fields ready for stacking\n    concat_sub['target_max'] = concat_sub.iloc[:, 1:].max(axis=1)\n    concat_sub['target_min'] = concat_sub.iloc[:, 1:].min(axis=1)\n    concat_sub['target_mean'] = concat_sub.iloc[:, 1:].mean(axis=1) # Not used but available if needed\n    concat_sub['target_median'] = concat_sub.iloc[:, 1:].median(axis=1) # Not used but available if needed\n    print(\" ***** 2/4 Get the data fields ready for stacking SUCCESSFUL *****\")\n    \n    # Set up cutoff threshold for lower and upper bounds\n    concat_sub['target_base'] = sub_base[column_names[1]]\n    concat_sub[column_names[1]] = np.where(np.all(concat_sub.iloc[:, 1:] > cutoff_lo, axis=1),\n                                    concat_sub['target_max'],\n                                    np.where(np.all(concat_sub.iloc[:, 1:] < cutoff_hi, axis=1),\n                                             concat_sub['target_min'],\n                                             concat_sub['target_base']))\n    print(\" ***** 3/4 Set up cutoff threshold for lower and upper bounds SUCCESSFUL *****\")\n    \n    # Generating Stacked dataframe\n    concat_sub[column_names].to_csv(output_path, index=False, float_format='%.12f')\n    print(\" ***** 4/4 Generating Stacked dataframe SUCCESSFUL *****\")\n    print(\" ***** COMPLETED *****\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"Stacking(input_folder = '../input/siim-isic-baseline-models/',\n         best_base = '../input/siim-isic-baseline-models/RESNET_0946.csv',\n         output_path = 'stacking.csv',\n         column_names = ['image_name', 'target'],\n         cutoff_lo = 0.85,\n         cutoff_hi = 0.17)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Ensemble","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def Ensemble(input_folder, \n             output_path, \n             method, \n             column_names, \n             sorted_files, \n             reverse = False):\n    \n    '''\n    To be tried on:\n        - different weak learners (models)\n        - several models for manual weightings\n    '''\n    \n    all_files = os.listdir(input_folder)\n    nb_files = len(all_files)\n    \n    # Warning\n    print(\"***** WARNING *****\\n\")\n    print(\"Your files must be written this way: model_score.csv:\")\n    print(\"    - Model without underscore, for example if you use EfficientNet do not write Eff_Net_075.csv but rather EffNet_075.csv\")\n    print(\"    - Score without comma, for example if you score 0.95 on XGB, the name can be XGB_095.csv\\n\")\n    print(\"About the score:\")\n    print(\"    - If the score has to be the lowest as possible, set reverse=True as argument\\n\")\n    \n    if (sorted_files == False) & (method in ['sum_of_integers', 'sum_of_squares']):\n        print(\"Arguments 'sum_of_integers' and 'sum_of_squares' might perform poorly as your files are not sorted\")\n        print(\"     - To sort them, change 'sorted_files' argument to 'True'\\n\")\n        \n    # Test compliancy of arguments\n    assert type(input_folder) == str, \"Wrong type\"\n    assert type(output_path) == str, \"Wrong type\"\n    assert len(column_names) == 2, \"Only two columns must be in column_names\"\n    assert type(column_names[0]) == str, \"Wrong type\"\n    assert type(column_names[1]) == str, \"Wrong type\"\n    assert method in ['mean', 'geometric_mean', 'sum_of_integers', 'sum_of_squares', 'weights'], 'Select a method among : mean, geometric_mean, sum_of_integers, sum_of_squares, weights.'\n    assert type(sorted_files) == bool, \"Wrong type\"\n    assert type(reverse) == bool, \"Wrong type\"\n    assert nb_files >= 1, 'Need at least two models for ensembling.'\n    \n    # Sorting models by performance\n    if sorted_files == True:\n        \n        # Sort files based on performance\n        ranking = [int(re.findall(r'\\_(\\d*)', file)[0]) for file in all_files]\n        dict_files = dict(zip(all_files, ranking))\n        sorted_dict = sorted(dict_files.items(), key=lambda x: x[1], reverse = reverse)\n        \n        assert len(all_files) == len([file[0] for file in sorted_dict]), \"Something went wrong with regex filtering\"\n        all_files = [file[0] for file in sorted_dict]\n        print(\" ***** Sorting models by performance SUCCESSFUL *****\")\n\n    # Create list of dataframes\n    DATAFRAMES = [pd.read_csv(input_folder + file) for file in all_files]\n    print(\" ***** 1/4 Create list of dataframes SUCCESSFUL *****\")\n\n    # Create the submission datdaframe initialized with first column\n    sub = pd.DataFrame()\n    sub[column_names[0]] = DATAFRAMES[0][column_names[0]]\n    print(\" ***** 2/4 Create the submission datdaframe SUCCESSFUL *****\")\n    \n    # Apply ensembling according to the method\n    if method == 'mean':\n        sub[column_names[1]] = np.mean([rankdata(df[column_names[1]], method='min') for df in DATAFRAMES], axis = 0)\n        \n    elif method == 'geometric_mean':\n        sub[column_names[1]] = np.exp(np.mean([rankdata(df[column_names[1]].apply(lambda x: np.log2(x)), method='min') for df in DATAFRAMES], axis = 0))\n        \n    elif method == 'sum_of_integers':        \n        constant = 1/(nb_files*(nb_files+1)/2)\n        sub[column_names[1]] = np.sum([(i+1)*rankdata(DATAFRAMES[i][column_names[1]], method='min') for i in range(nb_files)], axis = 0) * constant\n    \n    elif method == 'sum_of_squares':\n        constant = 1/((nb_files*(nb_files+1)*(2*nb_files+1))/6)\n        sub[column_names[1]] = np.sum([(i+1)*(i+1)*rankdata(DATAFRAMES[i][column_names[1]], method='min') for i in range(nb_files)], axis = 0) * constant\n    \n    elif method == 'weights':\n        # Type manually here your own weights\n        #print(all_files)\n        weights = [0.2, 0.35, 0.45]\n        assert len(weights) == nb_files, \"Length of weights doesn't fit with number of models to be ensembled\"\n        assert sum(weights) == 1, 'Sum of weights must be equal to 1'\n        sub[column_names[1]] = np.sum([weights[i]*rankdata(DATAFRAMES[i][column_names[1]], method='min') for i in range(nb_files)], axis = 0)\n        print('\\n')\n        for i in range(len(weights)):\n            print(f'    - Applied weight {weights[i]} to file {all_files[i]}')\n        print('\\n')\n            \n        \n    print(\" ***** 3/4 Apply ensembling according to the method SUCCESSFUL *****\")\n    sub.to_csv(output_path, index=False, float_format='%.12f')\n    print(\" ***** 4/4 Generating Ensembled dataframe SUCCESSFUL *****\")\n    print(\" ***** COMPLETED *****\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"Ensemble(input_folder = '../input/siim-isic-baseline-models/', \n         output_path = 'ensemble.csv',\n         method = 'weights',\n         column_names = ['image_name', 'target'], \n         sorted_files = True,\n         reverse = False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# References\n\n* https://towardsdatascience.com/ensemble-methods-bagging-boosting-and-stacking-c9214a10a205#:~:text=Combine%20weak%20learners&text=The%20ensemble%20model%20we%20obtain,said%20to%20be%20%E2%80%9Chomogeneous%E2%80%9D.&text=stacking%2C%20that%20often%20considers%20heterogeneous,the%20different%20weak%20models%20predictions\n\n* https://machinelearningmastery.com/stacking-ensemble-machine-learning-with-python/\n\n* https://www.youtube.com/watch?v=sBrQnqwMpvA","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"<hr>\n<br>\n<div align='justify'><font color=\"#353B47\" size=\"4\">Thank you for taking the time to read this notebook. I hope that I was able to answer your questions or your curiosity and that it was quite understandable. <u>any constructive comments are welcome</u>. They help me progress and motivate me to share better quality content. I am above all a passionate person who tries to advance my knowledge but also that of others. If you liked it, feel free to <u>upvote and share my work.</u> </font></div>\n<br>\n<div align='center'><font color=\"#353B47\" size=\"3\">Thank you and may passion guide you.</font></div>","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}