{"cells":[{"metadata":{},"cell_type":"markdown","source":"# EfficientNetBi - Melanoma Classification with Tensorflow/Keras\n\nThis kernel presents the pipeline to train the EfficientNet models using TPUs with Tensorflow/Keras.\n\nThe results of all models, from B0 to B7, are presented [here](https://www.kaggle.com/fredericods/from-efficientb0-to-b7-melanoma-classification/).\n\n**Training/Modeling Highlights:**\n- Input size: 256 x 256\n- Fine tuning EfficientNetBi with only a dense layer on the top\n- Stratified Group K-Fold Validation: imbalanced target distribution + not have same patient on train and validation set\n- Augmentations: random flip left-right and random flip up-down\n- Learning Rate Scheduler: adopting a learning rate ramp-up because fine-tuning a pre-trained model\n- Adam optimizer\n- Loss function: Binary Cross-Entropy Loss with label_smoothing\n- Epochs: 10\n- Model checkpoint: saving when best validation loss is achieved\n\n**References:**\n- https://www.kaggle.com/reighns/groupkfold-efficientbnet-and-augmentations\n- https://www.kaggle.com/jakubwasikowski/stratified-group-k-fold-cross-validation\n- https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96\n- https://www.kaggle.com/ajaykumar7778/melanoma-tpu-efficientnet-b5-dense-head\n- https://www.kaggle.com/khoongweihao/siim-isic-multiple-model-training-stacking\n\n## 1) Importing libraries and dataset","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"MODEL_NAME = \"EfficientNetB0\"\n# MODEL_NAME = \"EfficientNetB1\"\n# MODEL_NAME = \"EfficientNetB2\"\n# MODEL_NAME = \"EfficientNetB3\"\n# MODEL_NAME = \"EfficientNetB4\"\n# MODEL_NAME = \"EfficientNetB5\"\n# MODEL_NAME = \"EfficientNetB6\"\n# MODEL_NAME = \"EfficientNetB7\"","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import os\nimport re\nimport seaborn as sns\nimport numpy as np\nimport pandas as pd\nimport math\nimport random\nfrom collections import Counter, defaultdict\n\nfrom matplotlib import pyplot as plt\n\nfrom sklearn import metrics\n\nimport tensorflow as tf\nimport tensorflow.keras.layers as L\n\n!pip install -q efficientnet\nimport efficientnet.tfkeras as efn\n\nfrom kaggle_datasets import KaggleDatasets","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2) Preparing data and training","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### TPU config","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"try:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy()\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"AUTO = tf.data.experimental.AUTOTUNE\nignore_order = tf.data.Options()\nignore_order.experimental_deterministic = False","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Implementing Stratified Group K-fold","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def stratified_group_k_fold(X, y, groups, k, seed=None):\n    labels_num = np.max(y) + 1\n    y_counts_per_group = defaultdict(lambda: np.zeros(labels_num))\n    y_distr = Counter()\n    for label, g in zip(y, groups):\n        y_counts_per_group[g][label] += 1\n        y_distr[label] += 1\n\n    y_counts_per_fold = defaultdict(lambda: np.zeros(labels_num))\n    groups_per_fold = defaultdict(set)\n\n    def eval_y_counts_per_fold(y_counts, fold):\n        y_counts_per_fold[fold] += y_counts\n        std_per_label = []\n        for label in range(labels_num):\n            label_std = np.std([y_counts_per_fold[i][label] / y_distr[label] for i in range(k)])\n            std_per_label.append(label_std)\n        y_counts_per_fold[fold] -= y_counts\n        return np.mean(std_per_label)\n    \n    groups_and_y_counts = list(y_counts_per_group.items())\n    random.Random(0).shuffle(groups_and_y_counts)\n\n    for g, y_counts in sorted(groups_and_y_counts, key=lambda x: -np.std(x[1])):\n        best_fold = None\n        min_eval = None\n        for i in range(k):\n            fold_eval = eval_y_counts_per_fold(y_counts, i)\n            if min_eval is None or fold_eval < min_eval:\n                min_eval = fold_eval\n                best_fold = i\n        y_counts_per_fold[best_fold] += y_counts\n        groups_per_fold[best_fold].add(g)\n\n    all_groups = set(groups)\n    for i in range(k):\n        train_groups = all_groups - groups_per_fold[i]\n        test_groups = groups_per_fold[i]\n\n        train_indices = [i for i, g in enumerate(groups) if g in train_groups]\n        test_indices = [i for i, g in enumerate(groups) if g in test_groups]\n\n        yield train_indices, test_indices","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Preparing dataset","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Defining paths\nKAGGLE_PATH = \"/kaggle/input/siim-isic-melanoma-classification/\"\nIMG_PATH_TRAIN = \"/kaggle/input/siim-isic-melanoma-classification/jpeg/train/\"\nIMG_PATH_TEST = \"/kaggle/input/siim-isic-melanoma-classification/jpeg/test/\"\nGCS_PATH = KaggleDatasets().get_gcs_path(\"siim-isic-melanoma-classification\")\n\n# Importing train and test data\ndf_train = pd.read_csv(os.path.join(KAGGLE_PATH, \"train.csv\"))        \ndf_test = pd.read_csv(os.path.join(KAGGLE_PATH, \"test.csv\"))\n\n# Creating columns image_path on train and test data\ndf_train['image_path'] = df_train['image_name'].apply(lambda x: GCS_PATH + '/jpeg/train/' + x + '.jpg').values\ndf_test['image_path'] = df_test['image_name'].apply(lambda x: GCS_PATH + '/jpeg/test/' + x + '.jpg').values\n\n# Creating kfold column and train data using Stratified Group K-fold\ndf_train[\"kfold\"] = -1\ndf_train = df_train.sample(frac=1, random_state=0).reset_index(drop=True) # shuffle dataframe\ny = df_train.target.values\nfor fold_, (train_idx, test_idx) in enumerate(stratified_group_k_fold(df_train, df_train.target.values, df_train.patient_id.values, k=4)):\n    df_train.loc[test_idx, \"kfold\"] = fold_","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def decode_image(filename,label=None):\n    bits = tf.io.read_file(filename)\n    image = tf.image.decode_jpeg(bits, channels=3)\n    image = tf.cast(image, tf.float32) / 255.0\n    image = tf.image.resize(image, size = (IMG_SIZE, IMG_SIZE))\n    if label is None:\n        return image\n    else:\n        return image, label\n    \ndef data_augment(image, label=None):\n    image = tf.image.random_flip_left_right(image)\n    image = tf.image.random_flip_up_down(image)\n    if label is None:\n        return image\n    else:\n        return image, label\n    \ndef get_training_valid_dataset(kfold_i):\n    train_paths_fold_i = df_train[df_train['kfold'] != kfold_i]['image_path']\n    train_labels_fold_i = df_train[df_train['kfold'] != kfold_i]['target']\n    \n    valid_paths_fold_i = df_train[df_train['kfold'] == kfold_i]['image_path']\n    valid_labels_fold_i = df_train[df_train['kfold'] == kfold_i]['target']\n    \n    train_dataset = (\n        tf.data.Dataset\n        .from_tensor_slices((train_paths_fold_i, train_labels_fold_i))\n        .map(decode_image, num_parallel_calls=AUTO)\n        .map(data_augment, num_parallel_calls=AUTO)\n        .repeat()\n        .shuffle(1024)\n        .batch(BATCH_SIZE)\n        .prefetch(AUTO)\n    )\n    \n    valid_dataset = (\n        tf.data.Dataset\n        .from_tensor_slices((valid_paths_fold_i, valid_labels_fold_i))\n        .map(decode_image, num_parallel_calls=AUTO)\n        .batch(BATCH_SIZE)\n        .cache()\n        .prefetch(AUTO)\n    )\n    \n    return {'train': train_dataset, 'valid': valid_dataset}\n\n\ndef get_test_dataset():\n    test_paths = df_test['image_path']\n    \n    test_dataset = (\n        tf.data.Dataset\n        .from_tensor_slices(test_paths)\n        .map(decode_image, num_parallel_calls=AUTO)\n        .batch(BATCH_SIZE)\n    )\n    \n    return test_dataset","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3) Model Training","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Learning Rate Scheduler","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"LR_START = 0.00001\nLR_MAX = 0.00005*strategy.num_replicas_in_sync\nLR_MIN = 0.000001\nLR_RAMPUP_EPOCHS = 10\nLR_SUSTAIN_EPOCHS = 0\nLR_EXP_DECAY = 0.8\n\ndef lrfn(epoch):\n    if epoch < LR_RAMPUP_EPOCHS:\n        lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n    elif epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n        lr = LR_MAX\n    else:\n        lr = (LR_MAX - LR_MIN) * LR_EXP_DECAY**(epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS) + LR_MIN\n    return lr\n\nrng = [i for i in range(50)]\ny = [lrfn(x) for x in rng]\nplt.plot(rng, y);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### EfficientNetBi Model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_pretrained_model(model_name):\n    if model_name==\"EfficientNetB0\":\n        pretrained_model=efn.EfficientNetB0(include_top=False, input_shape=(IMG_SIZE, IMG_SIZE, 3), weights='imagenet')\n    if model_name==\"EfficientNetB1\":\n        pretrained_model=efn.EfficientNetB1(include_top=False, input_shape=(IMG_SIZE, IMG_SIZE, 3), weights='imagenet')\n    if model_name==\"EfficientNetB2\":\n        pretrained_model=efn.EfficientNetB2(include_top=False, input_shape=(IMG_SIZE, IMG_SIZE, 3), weights='imagenet')\n    if model_name==\"EfficientNetB3\":\n        pretrained_model=efn.EfficientNetB3(include_top=False, input_shape=(IMG_SIZE, IMG_SIZE, 3), weights='imagenet')\n    if model_name==\"EfficientNetB4\":\n        pretrained_model=efn.EfficientNetB4(include_top=False, input_shape=(IMG_SIZE, IMG_SIZE, 3), weights='imagenet')\n    if model_name==\"EfficientNetB5\":\n        pretrained_model=efn.EfficientNetB5(include_top=False, input_shape=(IMG_SIZE, IMG_SIZE, 3), weights='imagenet')\n    if model_name==\"EfficientNetB6\":\n        pretrained_model=efn.EfficientNetB6(include_top=False, input_shape=(IMG_SIZE, IMG_SIZE, 3), weights='imagenet')\n    if model_name==\"EfficientNetB7\":\n        pretrained_model=efn.EfficientNetB7(include_top=False, input_shape=(IMG_SIZE, IMG_SIZE, 3), weights='imagenet')\n    return pretrained_model ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Define model\ndef get_model():\n    with strategy.scope():\n        model = tf.keras.Sequential([\n            get_pretrained_model(MODEL_NAME),\n            L.GlobalAveragePooling2D(),\n            L.Dense(1, activation='sigmoid')\n        ])\n    \n        model.compile(\n            optimizer='adam',\n            loss = tf.keras.losses.BinaryCrossentropy(label_smoothing=0.1),\n            metrics=[tf.keras.metrics.AUC()]\n        )\n\n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Train Function","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"IMG_SIZE = 256\nBATCH_SIZE = 8 * strategy.num_replicas_in_sync\nSTEPS_PER_EPOCH = df_train[df_train.kfold !=0].shape[0] // BATCH_SIZE\n\ndef train_fold(kfold_i):\n    \n    my_callbacks = [\n        tf.keras.callbacks.LearningRateScheduler(lrfn, verbose = True),\n        #tf.keras.callbacks.EarlyStopping(patience=10, monitor=\"val_auc\", mode=\"max\"),\n        tf.keras.callbacks.ModelCheckpoint(filepath=MODEL_NAME + \"_fold_\" + str((kfold_i)) + \".h5\",save_best_only=True)\n    ]\n    \n    model_i = get_model()\n    dataset_i = get_training_valid_dataset(kfold_i)\n    \n    history_i = model_i.fit(\n        dataset_i['train'],\n        validation_data=dataset_i['valid'],\n        epochs=10,\n        callbacks=my_callbacks,\n        steps_per_epoch=STEPS_PER_EPOCH\n    )\n\n    df_history_i = pd.DataFrame(history_i.history)\n    df_history_i.to_csv(\"history_\" + MODEL_NAME + \"_fold_{}.csv\".format(kfold_i))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Prediction Function","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def save_predictions(fold_i):\n    with strategy.scope():\n        model_name_fold_i = MODEL_NAME + \"_fold_\" + str((fold_i)) + \".h5\"\n        model_fold_i = tf.keras.models.load_model(model_name_fold_i)\n        \n        probs_fold_i = model_fold_i.predict(get_test_dataset(),verbose = 1)\n        submit_fold_i = pd.read_csv(os.path.join(KAGGLE_PATH, \"sample_submission.csv\")) \n        submit_fold_i['target'] = probs_fold_i\n\n        submit_fold_i.to_csv(\"submit_\" + MODEL_NAME + \"_fold_{}.csv\".format(fold_i), index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Train Folds","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"#### Fold 0","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_fold(0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"save_predictions(0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Fold 1","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#train_fold(1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#save_predictions(1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Fold 2","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#train_fold(2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#save_predictions(2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Fold 3","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#train_fold(3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#save_predictions(3)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}