{"cells":[{"metadata":{},"cell_type":"markdown","source":"# From EfficientNetB0 to B7 - Melanoma Classification with Tensorflow/Keras\n\nThis kernel presents the results of training the EfficientNet models, from B0 to B7, using TPUs (yeah, it took a looooooong time to train it all).\n\nAnd the kernel I used to train all these models is [here](https://www.kaggle.com/fredericods/efficientnetbi-melanoma-classification-with-tf). \n\nThe main idea is to analyze the influence of the model size on its performance. And obviously, practice computer vision and deep learning skills. :)\n\nThe performance of the trained networks here is still far behind the top-score solutions, but there is a lot of room for improvement: tuning the learning rate, larger input size, larger networks, more complex augmentations, using metadata, ensembling etc.\n\nFeel free to criticize and suggest!\n\n**Training/Modeling Highlights:**\n- Image input size: 256 x 256\n- Fine tuning EfficientNet with only a dense layer on the top\n- Stratified Group 4-Fold Validation: imbalanced target distribution + not have same patient on train and validation set\n- Augmentations: random flip left-right and random flip up-down\n- Learning Rate Scheduler: adopting a learning rate ramp-up because fine-tuning a pre-trained model\n- Adam optimizer\n- Loss function: Binary Cross-Entropy Loss with label_smoothing\n- Epochs: 10\n- Model checkpoint: saving when best validation loss is achieved\n\n**References:**\n- https://www.kaggle.com/reighns/groupkfold-efficientbnet-and-augmentations\n- https://www.kaggle.com/jakubwasikowski/stratified-group-k-fold-cross-validation\n- https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96\n- https://www.kaggle.com/ajaykumar7778/melanoma-tpu-efficientnet-b5-dense-head\n- https://www.kaggle.com/khoongweihao/siim-isic-multiple-model-training-stacking","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 1) Importing libraries and dataset","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import os\nfrom os import listdir\nfrom os.path import isfile, join\n\nimport numpy as np\nimport pandas as pd\n\nfrom plotly.subplots import make_subplots\nimport plotly.graph_objects as go\n        \nDATA_PATH = '/kaggle/input/efficientbx-melanoma-classification-with-tf'\n\n# Importing Data\nhistory_files = [f for f in listdir(DATA_PATH) if isfile(join(DATA_PATH, f)) and f.split('_')[0] == 'history']\nsubmit_files = [f for f in listdir(DATA_PATH) if isfile(join(DATA_PATH, f)) and f.split('_')[0] == 'submit']","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Preprocessing Data for visualization (Performance x Model)\nlist_results = []\nfor file_name_i in history_files:\n    model_name_i = file_name_i[8:22]\n    fold_name_i = file_name_i[23:29]\n    df_i = pd.read_csv(os.path.join(DATA_PATH, file_name_i), index_col=0)\n    auc_i = df_i[df_i.val_loss == df_i.val_loss.min()]['val_auc'].iloc[0]\n    loss_i = df_i.val_loss.min()\n    list_results.append([model_name_i, fold_name_i, auc_i, loss_i])\ndf_results = pd.DataFrame(list_results, columns = ['Model', 'Fold', 'AUC', 'Loss'])\n\nfor model_name_i in df_results.Model.unique():\n    mean_auc_i = df_results.AUC[df_results.Model == model_name_i].mean()\n    mean_loss_i = df_results.Loss[df_results.Model == model_name_i].mean()\n    df_i = pd.DataFrame([[model_name_i, 'mean_fold', mean_auc_i, mean_loss_i]], columns = ['Model', 'Fold', 'AUC', 'Loss'])\n    df_results = pd.concat([df_results, df_i])\n    \ndf_results = df_results.sort_values(by = ['Model', 'Fold'], ascending = True).reset_index().drop(columns = ['index'])","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Preprocessing Data for visualization (Training History)\nmodel_names = ['EfficientNetB' + str(i) for i in range(8)]\nfold_names = ['fold_' + str(i) for i in range(4)]\ndict_df_history = {}\nfor model_i in model_names:\n    dict_df_history[model_i] = {}\n    for fold_i in fold_names:\n        file_name_i = 'history_' + model_i + '_' + fold_i + '.csv'\n        df_history_i = pd.read_csv(os.path.join(DATA_PATH, file_name_i))\n        df_history_i=df_history_i.rename(columns = {\n            'Unnamed: 0': 'Epoch',\n            'loss': 'Train Loss',\n            'auc': 'Train AUC',\n            'val_loss': 'Valid Loss',\n            'val_auc': 'Valid AUC',\n            'lr': 'Learning Rate'\n        }\n                           )\n        dict_df_history[model_i][fold_i] = df_history_i","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2) Visualizing Results","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"from plotly.subplots import make_subplots\nimport plotly.graph_objects as go\n\ndf_results_mean_fold = df_results[df_results.Fold=='mean_fold']\n\nfig = make_subplots(rows=2, cols=1)\n\nfig.append_trace(\n    go.Scatter(x=df_results_mean_fold.Model, y=df_results_mean_fold.AUC, name='AUC'),\n    row=1, col=1\n)\n\nfig.append_trace(\n    go.Scatter(x=df_results_mean_fold.Model, y=df_results_mean_fold.Loss, name='Loss'),\n    row=2, col=1\n)\n\n\nfig.update_layout(height=800, width=600, title_text=\"Performance x Model\")\nfig.update_yaxes(title_text=\"AUC\", row=1, col=1)\nfig.update_yaxes(title_text=\"Loss\", row=2, col=1)\n\nfor trace in fig['data']: \n    trace['showlegend'] = False\n        \nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Model Training History","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = make_subplots(rows=2, cols=2, subplot_titles=('Fold 0', 'Fold 1', 'Fold 2', 'Fold 3'))\n\n# Add Traces\nfor model_name_i in model_names:\n    for i, fold_i in enumerate(fold_names):\n        for set_i in ['Train', 'Valid']:\n            for metric_i in ['AUC', 'Loss']:\n                index_dict = {\n                    'fold_0': [1,1],\n                    'fold_1': [1,2],\n                    'fold_2': [2,1],\n                    'fold_3': [2,2]\n                }\n                color_dict = {\n                    'Train AUC': {'color': '#dba053'},\n                    'Valid AUC': {'color': '#6e53db'},\n                    'Train Loss': {'color': '#4cd489'},\n                    'Valid Loss': {'color': '#bf2e2e'}\n                }\n                fig.add_trace(\n                    go.Scatter(\n                        x=dict_df_history[model_name_i][fold_i]['Epoch'],\n                        y=dict_df_history[model_name_i][fold_i][set_i + ' ' + metric_i],\n                        name=set_i + ' ' + metric_i,\n                        #name=model_name_i + ' ' + fold_i + ' ' + set_i + ' ' + metric_i,\n                        #name=model_name_i[-2:] + fold_i[-1] + set_i + metric_i,\n                        line=color_dict[set_i + ' ' + metric_i],\n                        legendgroup=model_name_i[-2:] + ' ' + metric_i,\n                        showlegend=(i==0)\n                    ),\n                    row=index_dict[fold_i][0], col=index_dict[fold_i][1]\n                )\n\nlist_buttons = []\nfor i, model_name_i in enumerate(model_names):\n    visible_list_aux_i = [False]*128\n    visible_list_aux_i[i*16:i*16+16] = [True]*16\n    button_i = dict(\n        label=model_name_i[-2:],\n        method=\"update\",\n        args=[{\"visible\": visible_list_aux_i}])\n    list_buttons.append(button_i)                \n                \nfig.update_layout(\n    updatemenus=[\n        dict(\n            active=0,\n            buttons=list_buttons,\n            direction=\"right\",\n            pad={\"r\": 10, \"t\": 10},\n            showactive=True,\n            x=0.00,\n            xanchor=\"left\",\n            y=1.25,\n            yanchor=\"top\",\n            type = 'buttons'\n        )\n    ])\n\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3) Submit Files","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_submit = pd.read_csv('/kaggle/input/siim-isic-melanoma-classification/sample_submission.csv')\nfor model_name_i in model_names:\n    target_list = []\n    for fold_i in fold_names:\n        target_i = pd.read_csv(os.path.join(DATA_PATH, 'submit_' + model_name_i + '_' + fold_i + '.csv'))['target'].values\n        target_list.append(target_i)\n    df_submit_i = sample_submit.copy()\n    df_submit_i['target'] = np.mean(target_list, axis=0)\n    df_submit_i.to_csv('submit_' + model_name_i + '_mean_fold.csv', index=False)\n    print(model_name_i + ' - Mean Fold submit file saved')","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}