{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Introduction to GroupKFold","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"In this kernel, I will be performing **scikit-learn's** `GroupKFold`. Ideally, since the data is severely imbalanced, we should implement `Stratified GroupKFold`, but its not implemented in the existing libraries; if one is interested, you can refer to [the link](https://www.kaggle.com/jakubwasikowski/stratified-group-k-fold-cross-validation) here for this implementation.\n\nThe reason I think `GroupKFold` might be better than normal `KFold` is as follows: Quoting from **scikit-learn's** [website](https://scikit-learn.org/stable/modules/cross_validation.html), it says: \n\n**The Independent and identically distributed (i.i.d.) assumption is broken if the underlying generative process yield groups of dependent samples.**\n\n**Such a grouping of data is domain specific. An example would be when there is medical data collected from multiple patients, with multiple samples taken from each patient. And such data is likely to be dependent on the individual group. In our example, the patient id for each sample will be its group identifier.**\n\n**In this case we would like to know if a model trained on a particular set of groups generalizes well to the unseen groups. To measure this, we need to ensure that all the samples in the validation fold come from groups that are not represented at all in the paired training fold.**\n\n\n**GroupKFold is a variation of k-fold which ensures that the same group is not represented in both testing/validation and training sets. For example if the data is obtained from different subjects with several samples per-subject and if the model is flexible enough to learn from highly person specific features it could fail to generalize to new subjects. `GroupKFold` makes it possible to detect this kind of overfitting situations.**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Do note this is my first time implementing this, and there might be things that I overlooked; Please do point out if you find any errors in the pipeline or logic flow.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Installing Libraries","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install -q efficientnet","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport numpy as np\nimport cv2\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\nimport os \nimport re\nimport math\nfrom matplotlib import pyplot as plt\nfrom math import ceil\nfrom sklearn import metrics\nfrom sklearn.model_selection import GroupKFold\nfrom sklearn.model_selection import train_test_split\nfrom sklearn import model_selection\n\nimport tensorflow as tf\nimport tensorflow.keras.layers as L\n\nimport efficientnet.tfkeras as efn\n\n# Suppress warnings \nimport warnings\nwarnings.filterwarnings('ignore')\n\nfrom kaggle_datasets import KaggleDatasets\n\nfrom IPython.core.interactiveshell import InteractiveShell\nInteractiveShell.ast_node_interactivity = \"all\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**References:**\n1. [Parul's Awesome EDA kernel](https://www.kaggle.com/parulpandey/melanoma-classification-eda-starter)\n2. [Tarun's PlantPathology2020 awesome kernel](https://www.kaggle.com/tarunpaparaju/plant-pathology-2020-eda-models/notebook)\n3. [Wei Hao's awesome kernel](https://www.kaggle.com/khoongweihao/siim-isic-multiple-model-training-inference/input)\n4. [Abhishek Thakur's kernel](https://www.kaggle.com/abhishek/melanoma-detection-with-pytorch)\n5. [Chris's Kernel](https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96/data)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Defining the paths","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Defining data path\ntrain_images_dir = '../input/siim-isic-melanoma-classification/train/'\ntest_images_dir = '../input/siim-isic-melanoma-classification/test/'\ntrain_csv = '../input/siim-isic-melanoma-classification/train.csv'\ntest_csv  = '../input/siim-isic-melanoma-classification/test.csv'\nsample_submission = '../input/siim-isic-melanoma-classification/sample_submission.csv'\n\ntrain_df = pd.read_csv(train_csv)\ntest_df = pd.read_csv(test_csv)\n\ntrain_df.head()\n\nprint(\"The unique number of patiend_ids are {}\".format(train_df['patient_id'].nunique()))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the above dataframe, it is not immediately obvious that one unique patient may have multiple **image_names** attached to them because the dataframe is not sorted according to the `patiend_id`. We can use `train_df['patient_id'].nunique()` to check that there are 2056 unique `patient_ids` out of a whooping 33126 rows. This means that many patients have multiple images. And as I mentioned earlier, when we do `KFold` splitting, let's say there are 20 images for patient 1, we may have patient 1's data/images in both the training and validation set. For example, in the splitting process, there are 15 images of patient 1 in the training set, and there are 5 images in the validation set; then this may not be ideal since the model has already seen 15 of the images for patient 1 and can easily remember features that are **unique** to patient 1, and therefore predict well in the validation set for the same patient 1. Therefore, this cross validation method may give over optimistic results and fail to generalize well to more unseen images.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Splitting the dataset according to GroupKFold","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"group_by_patient_id = train_df.groupby(['patient_id', 'image_name']) \ngroup_by_patient_id.first()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As you can see here, it is indeed the case that one patient can have multiple images. We will do the splitting below.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"groups_by_patient_id_list = train_df['patient_id'].copy().tolist()\n# the below code should work better in fact\n# groups_by_patient_id_list = np.array(train_df['patient_id'].values)\n\ny_labels = train_df[\"target\"].values\n# x_train = train_df[[\"image_name\",\"patient_id\",\"sex\",\"age_approx\",\"anatom_site_general_challenge\"]]\n# y_train = train_df[[\"target\"]]\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here I created 5 folds and appended the 10 dataframes into a list. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"n_splits = 5\ngkf = GroupKFold(n_splits = 5)\n\nresult = []   \nfor train_idx, val_idx in gkf.split(train_df, y_labels, groups = groups_by_patient_id_list):\n    train_fold = train_df.iloc[train_idx]\n    val_fold = train_df.iloc[val_idx]\n    result.append((train_fold, val_fold))\n    \ntrain_fold_1, val_fold_1 = result[0][0],result[0][1]\ntrain_fold_2, val_fold_2 = result[1][0],result[1][1]\ntrain_fold_3, val_fold_3 = result[2][0],result[2][1]\ntrain_fold_4, val_fold_4 = result[3][0],result[3][1]\ntrain_fold_5, val_fold_5 = result[4][0],result[4][1]\n\n\n\n# just to check if it works as intended\nsample = train_fold_1.groupby(\"patient_id\")\nsample.get_group(\"IP_0147446\")\nsample.get_group(\"IP_0147446\").count()\n# sample2 = val_fold_1.groupby(\"patient_id\")\n# sample2.get_group(\"IP_0063782\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Modelling","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Detect hardware, return appropriate distribution strategy\ntry:\n    # TPU detection. No parameters necessary if TPU_NAME environment variable is\n    # set: this is always the case on Kaggle.\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    # Default distribution strategy in Tensorflow. Works on CPU and single GPU.\n    strategy = tf.distribute.get_strategy()\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# For tf.dataset\nAUTO = tf.data.experimental.AUTOTUNE\n\n# Data access\nGCS_PATH = KaggleDatasets().get_gcs_path('siim-isic-melanoma-classification')\n\n# Configuration\nBATCH_SIZE = 8 * strategy.num_replicas_in_sync\nimage_size = 256\nEPOCHS = 3","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`format_path_train` and `format_path_test` merely takes in an image name, and returns the path to the image.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def format_path_train(img_name):\n    return GCS_PATH + '/jpeg/train/' + img_name + '.jpg'\n\ndef format_path_test(img_name):\n    return GCS_PATH + '/jpeg/test/' + img_name + '.jpg'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As you can see here, we used `format_path_train` and `.apply` to get the image path.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_paths_fold_1 = train_fold_1.image_name.apply(format_path_train).values\nval_paths_fold_1 = val_fold_1.image_name.apply(format_path_train).values\n\ntrain_labels_fold_1 = train_fold_1.target.values\nval_labels_fold_1 = val_fold_1.target.values\n\ntest_paths = test_df.image_name.apply(format_path_test).values","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def decode_image(filename, label=None, image_size=(image_size, image_size)):\n    bits = tf.io.read_file(filename)\n    image = tf.image.decode_jpeg(bits, channels=3)\n    image = tf.cast(image, tf.float32) / 255.0\n    image = tf.image.resize(image, size = image_size)\n    \n    if label is None:\n        return image\n    else:\n        return image, label\n\n# def data_augment(image, label=None):\n#     image = tf.image.random_flip_left_right(image)\n#     image = tf.image.random_flip_up_down(image)\n    \n#     if label is None:\n#         return image\n#     else:\n#         return image, label\n\ndef data_augment(image, label=None):\n    image = tf.image.random_flip_left_right(image)\n    image = tf.image.random_flip_up_down(image)\n#     image = tf.image.random_saturation(image, lower = 1, upper = 3)\n#     image = tf.image.adjust_brightness(image, delta = 0.3)\n    image = tf.image.random_contrast(image, lower = 1, upper = 2)\n    if label is None:\n        return image\n    else:\n        return image, label","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Explanation of `decode_image`","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"To understand what the function `decode_image` does: we will use a sample filename to test it out.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_paths_fold_1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_filename = 'gs://kds-fbc9e41d779a5db41c5e7caf080daf3b4d85265f605fdbc4ce15cbfa/jpeg/train/ISIC_2637011.jpg' \nsample_label = 0\nimage_size = 256\n\n# 1. tf.io_read_file takes in a Tensor of type string and outputs a ensor of type string. \n#    Reads and outputs the entire contents of the input filename. \nbits = tf.io.read_file(sample_filename)\n\n# 2. Decode a JPEG-encoded image to a uint8 tensor. You can also use tf.io.decode_jpeg but according to \n#    tensorflow's website, it might be cleaner to use tf.image.decode_jpeg\nimage = tf.image.decode_jpeg(bits, channels=3)\n\nimage.shape  # outputs TensorShape([4000, 6000, 3])\n\n# 3. image = tf.cast(image, tf.float32) / 255.0 is easy to understand, it takes in \n#    an image, and cast the image into the data type you want. Here we also normalized by dividing by 255.\n\nimage = tf.cast(image, tf.float32) / 255.0\n\n\n# 4. image = tf.image.resize(image, image_size) is also easy to understand. We merely resize this image to the image_size we wish for.\n#    take note in our function defined above, the argument image_size is a tuple already. So we must pass in a tuple of our desired image_size.\nimage = tf.image.resize(image, size = (image_size, image_size))\n\nimage.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Explanation of `tf.data.Dataset`","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"According to the tensorflow website: The `tf.data.Dataset` API supports writing descriptive and efficient input pipelines. Dataset usage follows a common pattern:\n\n- Create a source dataset from your input data.\n- Apply dataset transformations to preprocess the data.\n- Iterate over the dataset and process the elements.\n\nIteration happens in a streaming fashion, so the full dataset does not need to fit into memory.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# example from tensorflow's website\nsample_dataset = tf.data.Dataset.from_tensor_slices([1, 2, 3])\nfor element in sample_dataset:\n    print(element)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Printing out the first element of `tf.data.Dataset.from_tensor_slices((train_paths_fold_1, train_labels_fold_1))` gives us **(<tf.Tensor: shape=(), dtype=string, numpy=b'gs://kds-c89313da1d85616eec461ab327fed61e1335defb486fb7729cf897b1/jpeg/train/ISIC_2637011.jpg'>, <tf.Tensor: shape=(), dtype=int64, numpy=0>)**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"As you can see, we printed out the first `data` in the `dataset`. The returned `data` is actually a tuple of length 2. Why length 2? Because the first element of the `data` contains the image's information, but currently its still stored as a String format. The second element of the `data` returns the label which in this case the label is 0 (non-malignant).","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"dataset = tf.data.Dataset.from_tensor_slices((train_paths_fold_1, train_labels_fold_1))\nfor data in dataset:\n    print(len(data))\n    print(data[0])\n    print(data[1])   \n    break","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Our next step is to decode the image using our functions defined earlier. As you can see, we used the `map` function and used our `decode_image` to make our image data into the \"Tensor Numpy Array Format\".","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"dataset = tf.data.Dataset.from_tensor_slices((train_paths_fold_1, train_labels_fold_1)).map(decode_image, num_parallel_calls=AUTO)\nfor data in dataset:\n    print(len(data))\n    print(data[0])\n    print(data[1])\n    break","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_dataset = tf.data.Dataset.from_tensor_slices([1, 2, 3])\nsample_dataset = sample_dataset.repeat(3)\nlist(sample_dataset)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We will explan `repeat()` and `batch()` together. When specified a batch_size,then `.batch(32)` will dictate 32 training examples and will undergo training. Using `.repeat()` we can specify the number of times we want the dataset to be iterated. If no parameter is passed it will loop forever, usually is good to just loop forever and directly control the number of epochs with a standard loop. [References](https://towardsdatascience.com/how-to-use-dataset-in-tensorflow-c758ef9e4428)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# dataset = tf.data.Dataset.from_tensor_slices((train_paths_fold_1, train_labels_fold_1)).map(decode_image, num_parallel_calls=AUTO).repeat()\n# # for data in dataset:\n# #     print(len(data))\n# #     print(data[0])\n# #     print(data[1])\n# #     break","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# dataset = tf.data.Dataset.from_tensor_slices((train_paths_fold_1, train_labels_fold_1)).map(decode_image, num_parallel_calls=AUTO).repeat().batch(32)\n# # here it returns 32 images and its labels, because we specified our batch size to be 32! \n# for data in dataset:\n#     print(data)\n#     break","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Explanation of Data Augment","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Data Augmentation, as always, is helpful in training Neural Networks. In this particular competition, I believe tweaking the shades of the skin by using contrast, saturation and brightness etc may be helpful to generalize. Of course, we will include the good old horizontal and vertical flip as well. Below is a brief visualization of the augmentations that one can use in this pipeline.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"image_folder_path = '../input/siim-isic-melanoma-classification/jpeg/train/'\nchosen_image = cv2.imread(os.path.join(image_folder_path, \"ISIC_0074542.jpg\"))[:,:,::-1]\nplt.imshow(chosen_image)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"horizontal_flipped = tf.image.flip_left_right(chosen_image)\nvertically_flipped = tf.image.flip_up_down(chosen_image)\nadjusted_saturation = tf.image.adjust_saturation(chosen_image, saturation_factor = 2)\nadjusted_brightness = tf.image.adjust_brightness(chosen_image, delta = 0.3)\nadjusted_contrast = tf.image.adjust_contrast(chosen_image, contrast_factor = 2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_multiple_img(img_matrix_list, title_list, ncols, main_title=\"\"):\n    fig, myaxes = plt.subplots(figsize=(20, 15), nrows=ceil(len(img_matrix_list) / ncols), ncols=ncols, squeeze=False)\n    fig.suptitle(main_title, fontsize = 30)\n    fig.subplots_adjust(wspace=0.3)\n    fig.subplots_adjust(hspace=0.3)\n    for i, (img, title) in enumerate(zip(img_matrix_list, title_list)):\n        myaxes[i // ncols][i % ncols].imshow(img)\n        myaxes[i // ncols][i % ncols].set_title(title, fontsize=15)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"img_matrix_list = [chosen_image,horizontal_flipped,vertically_flipped,adjusted_saturation,adjusted_brightness,adjusted_contrast]\ntitle_list = [\"Original\", \"HorizontalFlipped\", \"VerticallyFlipped\", \"Saturated\",\"Brightness\",\"Contrast\"]\nplot_multiple_img(img_matrix_list, title_list, ncols = 3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Defining the dataset","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# shuffle() (if used) should be called before batch() - we want to shuffle records not batches.\ntrain_dataset_fold_1 = (\n    tf.data.Dataset\n    .from_tensor_slices((train_paths_fold_1, train_labels_fold_1))\n    .map(decode_image, num_parallel_calls=AUTO)\n    .map(data_augment, num_parallel_calls=AUTO)\n    .repeat()\n    .shuffle(512)\n    .batch(BATCH_SIZE)\n    .prefetch(AUTO))\n\n# Generally we don't shuffle a test/val set at all - \n# only the training set (We evaluate using the entire test set anyway, right? So why shuffle?).\n# https://stackoverflow.com/questions/56944856/tensorflow-dataset-questions-about-shuffle-batch-and-repeat\n# https://stackoverflow.com/questions/49915925/output-differences-when-changing-order-of-batch-shuffle-and-repeat\nvalid_dataset_fold_1 = (\n    tf.data.Dataset\n    .from_tensor_slices((val_paths_fold_1, val_labels_fold_1))\n    .map(decode_image, num_parallel_calls=AUTO)\n    .batch(BATCH_SIZE)\n    .cache()\n    .prefetch(AUTO))\n\ntest_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices(test_paths)\n    .map(decode_image, num_parallel_calls=AUTO)\n    .batch(BATCH_SIZE))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Defining our Learning Rate Scheduler function.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_lrfn(lr_start=0.00001, lr_max=0.00005, \n               lr_min=0.00001, lr_rampup_epochs=5, \n               lr_sustain_epochs=0, lr_exp_decay=.8):\n    lr_max = lr_max * strategy.num_replicas_in_sync\n\n    def lrfn(epoch):\n        if epoch < lr_rampup_epochs:\n            lr = (lr_max - lr_start) / lr_rampup_epochs * epoch + lr_start\n        elif epoch < lr_rampup_epochs + lr_sustain_epochs:\n            lr = lr_max\n        else:\n            lr = (lr_max - lr_min) *\\\n                 lr_exp_decay**(epoch - lr_rampup_epochs\\\n                                - lr_sustain_epochs) + lr_min\n        return lr\n    return lrfn\n\nlrfn = build_lrfn()\nlr_schedule = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose=1)\nrng = [i for i in range(25 if EPOCHS<25 else EPOCHS)]\ny = [lrfn(x) for x in rng]\nplt.plot(rng, y)\nprint(\"Learning rate schedule: {:.3g} to {:.3g} to {:.3g}\".format(y[0], max(y), y[-1]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Model CheckPoint","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"STEPS_PER_EPOCH = train_labels_fold_1.shape[0] // BATCH_SIZE\nmodel_checkpoint = tf.keras.callbacks.ModelCheckpoint('GroupKFold.h5', monitor='val_loss', verbose=2, save_best_only=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Define Model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_model():\n    with strategy.scope():\n        model = tf.keras.Sequential([\n            efn.EfficientNetB3(\n                input_shape=(256,256, 3),\n                weights=\"imagenet\",\n                include_top=False\n            ),\n            L.GlobalAveragePooling2D(),\n            L.Dense(1, activation='sigmoid')\n        ])\n        model.compile(\n            optimizer='adam',\n            loss = 'binary_crossentropy',\n            metrics=[tf.keras.metrics.AUC()])\n    \n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Fold 1","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"The code below is commented out as it was already defined earlier. But for the sake of completeness of each fold, I will include it below to make all folds look the same.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# train_paths_fold_1 = train_fold_1.image_name.apply(format_path_train).values\n# val_paths_fold_1 = val_fold_1.image_name.apply(format_path_train).values\n\n# train_labels_fold_1 = train_fold_1.target.values\n# val_labels_fold_1 = val_fold_1.target.values\n\n# train_dataset_fold_1 = (\n#     tf.data.Dataset\n#     .from_tensor_slices((train_paths_fold_1, train_labels_fold_1))\n#     .map(decode_image, num_parallel_calls=AUTO)\n#     .map(data_augment, num_parallel_calls=AUTO)\n#     .repeat()\n#     .shuffle(512)\n#     .batch(BATCH_SIZE)\n#     .prefetch(AUTO))\n\n# valid_dataset_fold_1 = (\n#     tf.data.Dataset\n#     .from_tensor_slices((val_paths_fold_1, val_labels_fold_1))\n#     .map(decode_image, num_parallel_calls=AUTO)\n#     .batch(BATCH_SIZE)\n#     .cache()\n#     .prefetch(AUTO))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"model_fold_1 = get_model()\nhistory_1 = model_fold_1.fit(train_dataset_fold_1,\n                    epochs=EPOCHS,\n                    callbacks=[model_checkpoint,lr_schedule],\n                    steps_per_epoch=STEPS_PER_EPOCH,\n                    validation_data=valid_dataset_fold_1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"probs_fold_1 = model_fold_1.predict(test_dataset,verbose = 1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Fold 2","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_paths_fold_2 = train_fold_2.image_name.apply(format_path_train).values\nval_paths_fold_2 = val_fold_2.image_name.apply(format_path_train).values\n\ntrain_labels_fold_2 = train_fold_2.target.values\nval_labels_fold_2 = val_fold_2.target.values\n\ntrain_dataset_fold_2 = (\n    tf.data.Dataset\n    .from_tensor_slices((train_paths_fold_2, train_labels_fold_2))\n    .map(decode_image, num_parallel_calls=AUTO)\n    .map(data_augment, num_parallel_calls=AUTO)\n    .repeat()\n    .shuffle(512)\n    .batch(BATCH_SIZE)\n    .prefetch(AUTO))\n\nvalid_dataset_fold_2 = (\n    tf.data.Dataset\n    .from_tensor_slices((val_paths_fold_2, val_labels_fold_2))\n    .map(decode_image, num_parallel_calls=AUTO)\n    .batch(BATCH_SIZE)\n    .cache()\n    .prefetch(AUTO))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model_fold_2 = get_model()\nhistory_2 = model_fold_2.fit(train_dataset_fold_2,\n                    epochs=EPOCHS,\n                    callbacks=[model_checkpoint,lr_schedule],\n                    steps_per_epoch=STEPS_PER_EPOCH,\n                    validation_data=valid_dataset_fold_2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"probs_fold_2 = model_fold_2.predict(test_dataset,verbose = 1) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Fold 3","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_paths_fold_3 = train_fold_3.image_name.apply(format_path_train).values\nval_paths_fold_3 = val_fold_3.image_name.apply(format_path_train).values\n\ntrain_labels_fold_3 = train_fold_3.target.values\nval_labels_fold_3 = val_fold_3.target.values\n\ntrain_dataset_fold_3 = (\n    tf.data.Dataset\n    .from_tensor_slices((train_paths_fold_3, train_labels_fold_3))\n    .map(decode_image, num_parallel_calls=AUTO)\n    .map(data_augment, num_parallel_calls=AUTO)\n    .repeat()\n    .shuffle(512)\n    .batch(BATCH_SIZE)\n    .prefetch(AUTO))\n\nvalid_dataset_fold_3 = (\n    tf.data.Dataset\n    .from_tensor_slices((val_paths_fold_3, val_labels_fold_3))\n    .map(decode_image, num_parallel_calls=AUTO)\n    .batch(BATCH_SIZE)\n    .cache()\n    .prefetch(AUTO))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model_fold_3 = get_model()\nhistory_3 = model_fold_3.fit(train_dataset_fold_3,\n                    epochs=EPOCHS,\n                    callbacks=[model_checkpoint,lr_schedule],\n                    steps_per_epoch=STEPS_PER_EPOCH,\n                    validation_data=valid_dataset_fold_3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"probs_fold_3 = model_fold_3.predict(test_dataset,verbose = 1) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Fold 4","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_paths_fold_4 = train_fold_4.image_name.apply(format_path_train).values\nval_paths_fold_4 = val_fold_4.image_name.apply(format_path_train).values\n\ntrain_labels_fold_4 = train_fold_4.target.values\nval_labels_fold_4 = val_fold_4.target.values\n\ntrain_dataset_fold_4 = (\n    tf.data.Dataset\n    .from_tensor_slices((train_paths_fold_4, train_labels_fold_4))\n    .map(decode_image, num_parallel_calls=AUTO)\n    .map(data_augment, num_parallel_calls=AUTO)\n    .repeat()\n    .shuffle(512)\n    .batch(BATCH_SIZE)\n    .prefetch(AUTO))\n\nvalid_dataset_fold_4 = (\n    tf.data.Dataset\n    .from_tensor_slices((val_paths_fold_4, val_labels_fold_4))\n    .map(decode_image, num_parallel_calls=AUTO)\n    .batch(BATCH_SIZE)\n    .cache()\n    .prefetch(AUTO))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model_fold_4 = get_model()\nhistory_4 = model_fold_4.fit(train_dataset_fold_4,\n                    epochs=EPOCHS,\n                    callbacks=[model_checkpoint,lr_schedule],\n                    steps_per_epoch=STEPS_PER_EPOCH,\n                    validation_data=valid_dataset_fold_4)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"probs_fold_4 = model_fold_4.predict(test_dataset,verbose = 1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Fold 5","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_paths_fold_5 = train_fold_5.image_name.apply(format_path_train).values\nval_paths_fold_5 = val_fold_5.image_name.apply(format_path_train).values\n\ntrain_labels_fold_5 = train_fold_5.target.values\nval_labels_fold_5 = val_fold_5.target.values\n\ntrain_dataset_fold_5 = (\n    tf.data.Dataset\n    .from_tensor_slices((train_paths_fold_5, train_labels_fold_5))\n    .map(decode_image, num_parallel_calls=AUTO)\n    .map(data_augment, num_parallel_calls=AUTO)\n    .repeat()\n    .shuffle(512)\n    .batch(BATCH_SIZE)\n    .prefetch(AUTO))\n\nvalid_dataset_fold_5 = (\n    tf.data.Dataset\n    .from_tensor_slices((val_paths_fold_5, val_labels_fold_5))\n    .map(decode_image, num_parallel_calls=AUTO)\n    .batch(BATCH_SIZE)\n    .cache()\n    .prefetch(AUTO))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model_fold_5 = get_model()\nhistory_5 = model_fold_5.fit(train_dataset_fold_5,\n                    epochs=EPOCHS,\n                    callbacks=[model_checkpoint,lr_schedule],\n                    steps_per_epoch=STEPS_PER_EPOCH,\n                    validation_data=valid_dataset_fold_5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"probs_fold_5 = model_fold_5.predict(test_dataset,verbose = 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_submission = '../input/siim-isic-melanoma-classification/sample_submission.csv'\nsubmission = pd.read_csv(sample_submission)\nsubmission['target'] = probs_fold_5   \nsubmission.head(20)\nsubmission.to_csv('submission_contrast_group_k_fold_5.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## TODO\nPlot AUC-ROC curve for validation and see confusion matrix, can see if we are having a lot of false negatives. or is the model just blindly predicting 0.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Submission for GroupKFold","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_submission = '../input/siim-isic-melanoma-classification/sample_submission.csv'\nsubmission = pd.read_csv(sample_submission)\nfold1 = pd.read_csv(\"../input/ensemble5folds/submission_group_k_fold_1.csv\")\nfold2 = pd.read_csv(\"../input/ensemble5folds/submission_group_k_fold_2.csv\")\nfold3 = pd.read_csv(\"../input/ensemble5folds/submission_group_k_fold_3.csv\")\nfold4 = pd.read_csv(\"../input/ensemble5folds/submission_group_k_fold_4.csv\")\nfold5 = pd.read_csv(\"../input/ensemble5folds/submission_group_k_fold_5.csv\")\nensembled = (fold1['target'] + fold2['target']  + fold3['target'] + fold4['target'] + fold5['target'])/5\nsubmission['target'] = ensembled\n\n# ensembled = (probs_fold_1 + probs_fold_2 + probs_fold_3 + probs_fold_4 + probs_fold_5)/5\n# submission['target'] = ensembled\nsubmission.head(20)\n#submitting to csv\nsubmission.to_csv('ensembled.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I got an LB score of 0.908 after ensembling.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Stratified GroupKFold","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**References:**\n\n[1. https://www.kaggle.com/jakubwasikowski/stratified-group-k-fold-cross-validation](https://www.kaggle.com/jakubwasikowski/stratified-group-k-fold-cross-validation)\n\n[2. https://www.kaggle.com/graf10a/siim-stratified-groupkfold-5-folds](https://www.kaggle.com/graf10a/siim-stratified-groupkfold-5-folds)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nimport random\nimport pandas as pd\nfrom collections import Counter, defaultdict\nfrom tqdm import tqdm\n\ndef stratified_group_k_fold(X, y, groups, k, seed=None):\n    \"\"\" https://www.kaggle.com/jakubwasikowski/stratified-group-k-fold-cross-validation \"\"\"\n    labels_num = np.max(y) + 1\n    y_counts_per_group = defaultdict(lambda: np.zeros(labels_num))\n    y_distr = Counter()\n    for label, g in zip(y, groups):\n        y_counts_per_group[g][label] += 1\n        y_distr[label] += 1\n\n    y_counts_per_fold = defaultdict(lambda: np.zeros(labels_num))\n    groups_per_fold = defaultdict(set)\n\n    def eval_y_counts_per_fold(y_counts, fold):\n        y_counts_per_fold[fold] += y_counts\n        std_per_label = []\n        for label in range(labels_num):\n            label_std = np.std([y_counts_per_fold[i][label] / y_distr[label] for i in range(k)])\n            std_per_label.append(label_std)\n        y_counts_per_fold[fold] -= y_counts\n        return np.mean(std_per_label)\n    \n    groups_and_y_counts = list(y_counts_per_group.items())\n    random.Random(seed).shuffle(groups_and_y_counts)\n\n    for g, y_counts in tqdm(sorted(groups_and_y_counts, key=lambda x: -np.std(x[1])), total=len(groups_and_y_counts)):\n        best_fold = None\n        min_eval = None\n        for i in range(k):\n            fold_eval = eval_y_counts_per_fold(y_counts, i)\n            if min_eval is None or fold_eval < min_eval:\n                min_eval = fold_eval\n                best_fold = i\n        y_counts_per_fold[best_fold] += y_counts\n        groups_per_fold[best_fold].add(g)\n\n    all_groups = set(groups)\n    for i in range(k):\n        train_groups = all_groups - groups_per_fold[i]\n        test_groups = groups_per_fold[i]\n\n        train_indices = [i for i, g in enumerate(groups) if g in train_groups]\n        test_indices = [i for i, g in enumerate(groups) if g in test_groups]\n\n        yield train_indices, test_indices","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"group_by_patient_id_array = np.array(train_df['patient_id'].values)\ny_labels = train_df[\"target\"].values","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"skf = stratified_group_k_fold(X=train_df, y=y_labels, groups=group_by_patient_id_array, k=5, seed=42)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_distribution(y_vals):\n        y_distr = Counter(y_vals)\n        y_vals_sum = sum(y_distr.values())\n        return [f'{y_distr[i] / y_vals_sum:.5%}' for i in range(np.max(y_vals) + 1)]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"distrs = [get_distribution(y_labels)]\nindex = ['training set']\n\nfor fold_ind, (dev_ind, val_ind) in enumerate(skf, 1):\n    dev_y, val_y = y_labels[dev_ind], y_labels[val_ind]\n    dev_groups, val_groups = group_by_patient_id_array[dev_ind], group_by_patient_id_array[val_ind]\n    # making sure that train and validation group do not overlap:\n    assert len(set(dev_groups) & set(val_groups)) == 0\n    \n    distrs.append(get_distribution(dev_y))\n    index.append(f'training set - fold {fold_ind}')\n    distrs.append(get_distribution(val_y))\n    index.append(f'validation set - fold {fold_ind}')\n\ndisplay('Distribution per class:')\npd.DataFrame(distrs, index=index, columns=[f'Label {l}' for l in range(np.max(y_labels) + 1)])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Each fold has almost the same percentages of 0's and 1's and this code given above allows the fact that unique `patient_id` values do not overlap between different folds.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df = train_df.copy()\ndf['fold'] = -1\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# somehow you need to redefine this skf line here for the .loc to work\nskf = stratified_group_k_fold(X=train_df, y=y_labels, groups=group_by_patient_id_array, k=5, seed=42)\nfor fold_number, (train_idx, val_idx) in enumerate(skf):\n    df.loc[val_idx, \"fold\"] = fold_number\n    \ndf.to_csv(\"sgkfold.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Training and Modelling","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Here I conveniently took the idea from [the grandmaster Abhishek Thakur](https://www.kaggle.com/abhishek/melanoma-detection-with-pytorch/data) way to modularize our training process.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Detect hardware, return appropriate distribution strategy\ntry:\n    # TPU detection. No parameters necessary if TPU_NAME environment variable is\n    # set: this is always the case on Kaggle.\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    # Default distribution strategy in Tensorflow. Works on CPU and single GPU.\n    strategy = tf.distribute.get_strategy()\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"GCS_PATH = KaggleDatasets().get_gcs_path('siim-isic-melanoma-classification')\n\ndef format_path_train(img_name):\n    return GCS_PATH + '/jpeg/train/' + img_name + '.jpg'\n\ndef format_path_test(img_name):\n    return GCS_PATH + '/jpeg/test/' + img_name + '.jpg'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"image_size = 256\n\ndef decode_image(filename, label=None, image_size=(image_size, image_size)):\n    bits = tf.io.read_file(filename)\n    image = tf.image.decode_jpeg(bits, channels=3)\n    image = tf.cast(image, tf.float32) / 255.0\n    image = tf.image.resize(image, size = image_size) \n\n    if label is None:\n        return image\n    else:\n        return image, label\n\ndef data_augment(image, label=None):\n    image = tf.image.random_flip_left_right(image)\n    image = tf.image.random_flip_up_down(image)\n\n    if label is None:\n        return image\n    else:\n        return image, label","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def train(fold_number):\n    training_data_path = '../input/siim-isic-melanoma-classification/train/'\n    df = pd.read_csv(\"/kaggle/working/sgkfold.csv\")\n    df_train = df[df.fold != fold_number].reset_index(drop=True)\n    df_valid = df[df.fold == fold_number].reset_index(drop=True)\n    df_train_path = df_train.image_name.apply(format_path_train).values\n    df_val_path   = df_valid.image_name.apply(format_path_train).values\n    df_train_labels = df_train.target.values\n    df_val_labels   = df_valid.target.values\n    \n    AUTO = tf.data.experimental.AUTOTUNE\n    # For tf.dataset\n    BATCH_SIZE = 8 * strategy.num_replicas_in_sync\n    EPOCHS = 3    \n    \n    train_dataset = (tf.data.Dataset\n    .from_tensor_slices((df_train_path, df_train_labels))\n    .map(decode_image, num_parallel_calls=AUTO)\n    .map(data_augment, num_parallel_calls=AUTO)\n    .repeat()\n    .shuffle(512)\n    .batch(BATCH_SIZE)\n    .prefetch(AUTO))\n    \n    valid_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices((df_val_path, df_val_labels))\n    .map(decode_image, num_parallel_calls=AUTO)\n    .batch(BATCH_SIZE)\n    .cache()\n    .prefetch(AUTO))\n    \n    def build_lrfn(lr_start=0.00001, lr_max=0.00005, \n                   lr_min=0.00001, lr_rampup_epochs=5, \n                   lr_sustain_epochs=0, lr_exp_decay=.8):\n        lr_max = lr_max * strategy.num_replicas_in_sync\n\n        def lrfn(epoch):\n            if epoch < lr_rampup_epochs:\n                lr = (lr_max - lr_start) / lr_rampup_epochs * epoch + lr_start\n            elif epoch < lr_rampup_epochs + lr_sustain_epochs:\n                lr = lr_max\n            else:\n                lr = (lr_max - lr_min) *\\\n                     lr_exp_decay**(epoch - lr_rampup_epochs\\\n                                    - lr_sustain_epochs) + lr_min\n            return lr\n        return lrfn    \n    \n    lrfn = build_lrfn()\n    STEPS_PER_EPOCH = df_train_labels.shape[0] // BATCH_SIZE\n    lr_schedule = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose=1)\n    model_checkpoint = tf.keras.callbacks.ModelCheckpoint('StratifiedGroupKFold.h5', monitor='val_loss', verbose=2, save_best_only=True)\n    \n    \n    with strategy.scope():\n        model = tf.keras.Sequential([\n            efn.EfficientNetB3(\n                input_shape=(256,256, 3),\n                weights=\"imagenet\",\n                include_top=False\n            ),\n            L.GlobalAveragePooling2D(),\n            L.Dense(1, activation='sigmoid')\n        ])\n        model.compile(\n            optimizer='adam',\n            loss = 'binary_crossentropy',\n            metrics=[tf.keras.metrics.AUC()])\n    \n    history = model.fit(train_dataset,\n                    epochs=EPOCHS,\n                    callbacks=[model_checkpoint,lr_schedule],\n                    steps_per_epoch=STEPS_PER_EPOCH,\n                    validation_data=valid_dataset)\n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Test Paths","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"test_paths = test_df.image_name.apply(format_path_test).values\nAUTO = tf.data.experimental.AUTOTUNE\nBATCH_SIZE = 8 * strategy.num_replicas_in_sync\nEPOCHS = 3  \n\ntest_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices(test_paths)\n    .map(decode_image, num_parallel_calls=AUTO)\n    .batch(BATCH_SIZE))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Training using Stratified GroupKFold","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fold_1 = train(0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"idling is not me.\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"probs_fold_1 = fold_1.predict(test_dataset, verbose = 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_submission = '../input/siim-isic-melanoma-classification/sample_submission.csv'\nsubmission = pd.read_csv(sample_submission)\nsubmission['target'] = probs_fold_1  \nsubmission.head(20)\nsubmission.to_csv('submission_stratified_group_k_fold_1.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Work in progress...**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fold_2 = train(1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"probs_fold_2 = fold_2.predict(test_dataset, verbose = 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fold_3 = train(2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"probs_fold_3 = fold_3.predict(test_dataset, verbose = 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fold_4 = train(3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"probs_fold_4 = fold_4.predict(test_dataset, verbose = 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fold_5 = train(4)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"probs_fold_5 = fold_5.predict(test_dataset, verbose = 1)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}