{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Melanoma Detection:\nGenerally in any medical image diagnosis Machine Learning problems, the number of positive labelled data will be less compared to negative labelled data since the number of people suffering from the disease will be less compared to number of people tested.It is no different in our current dataset.\n\nThe number of images corresponding to benign tumours is 98% which leads to huge Class Imbalance Problem.\n\nThere are various techniques for handling Class Imbalance.The one used is this kernel is ***UnderSampling***.\nUnderSampling in simple terms can be thought of as reducing the number of data points corresponding to the class which has significantly more data points in a class imbalance scenario\n\n![](http://)\n\n\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Installing Necessary Packages\n\n!pip install efficientnet\n!pip install sweetviz\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import os\nimport albumentations\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nfrom sklearn import metrics\nfrom sklearn import model_selection\nimport tensorflow as tf\nimport matplotlib.pyplot as plt\nimport cv2\nimport efficientnet.tfkeras as efn \nimport tensorflow.keras.layers as L\nfrom tensorflow.keras.models import Model\nfrom tensorflow.keras import backend as K\nfrom tensorflow.keras.utils import Sequence\nfrom tensorflow.keras.callbacks import (ModelCheckpoint, LearningRateScheduler,\n                                        EarlyStopping, ReduceLROnPlateau, CSVLogger)\nimport math\nimport sweetviz as sv","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create a dataframe out of train csv file\ndf = pd.read_csv(\"../input/siim-isic-melanoma-classification/train.csv\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Powerful Visualizations using sweetviz library","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Recently came across a Python Library called \"sweetviz\" which helps for basic EDA in just two lines of code!!!!!!\nIt generates a html report with interactive visualizations.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"my_report = sv.analyze(df, target_feat = 'target')\nmy_report.show_html(\"report.html\") # Default arguments will generate to \"SWEETVIZ_REPORT.html\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Just Hover over the feature in the html report, a graph would be shown in the extreme right [Need to open the notebook]","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"# from IPython.display import IFrame\n\n# IFrame(src='report.html', width=2000, height=1000)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df['benign_malignant'].value_counts()\ndf['target'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.countplot(x = 'target', data = df)\nsns.countplot(x = 'benign_malignant', data = df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10,10))\nsns.countplot(x = 'anatom_site_general_challenge', hue = 'target', data = df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## UnderSampling","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"On looking into the dataset it can be noted that for the same person[Patient ID] and for the same region of the body[anatom_site_general_challenge] , there are multiple Images. Only one image per person per anatomy region only is used, the rest all are dropped for benign cases.The malignant datapoints are not touched.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#Filtering benign and Malignant datapoints\n\ndf_malignant = df[df['target'] == 1]\ndf_benign = df[df['target'] == 0]\n\ndf_malignant = df_malignant.sample(frac=1).reset_index(drop=True)\ndf_benign = df_benign.sample(frac=1).reset_index(drop=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Dropping data points for benign cases\n\ndf_benign = df_benign.drop_duplicates(subset=['patient_id','anatom_site_general_challenge'], keep = \"first\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Concatenating the data frame\n\ndf = pd.concat([df_malignant, df_benign]).reset_index(drop = True)\ndf = df.sample(frac=1).reset_index(drop=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df.tail()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see from the counts below that the ratio is now close to 90:10 which is much better than 98:2\nAlso with stratified KFold technique the same ratio can be maintained in each folds","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df['target'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Initializing the parameters\n\nbatch_size = 16\ntrain_dir = \"../input/melanoma-merged-external-data-512x512-jpeg/512x512-dataset-melanoma/512x512-dataset-melanoma/\"\nIMG_HEIGHT = 512\nIMG_WIDTH = 512","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Stratified KFold samples [Inspired from Abhishek Thakur's Kernel]","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Stratified KFold samples\n\n\ndf = df.sample(frac=1).reset_index(drop=True)\nn_splits = 5\nfrom sklearn.model_selection import StratifiedKFold\nkf = StratifiedKFold(n_splits)\nfor fold, (_, val_ind) in enumerate(kf.split(X=df, y=df.target.values)):\n    df.loc[val_ind, 'fold'] = fold\n\ndf.to_csv(\"train_fold.csv\", index=False)\ndf = df.sample(frac=1).reset_index(drop=True) # shuffling ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def data_augmentor(image):\n    \n    '''\n    Function which perfoms certain random operations and returns the augmented Image\n    \n    '''\n    \n    image = tf.keras.preprocessing.image.random_shift(image,0.4,0.4)\n    image = tf.keras.preprocessing.image.random_rotation(image,20)\n    image = tf.keras.preprocessing.image.random_zoom(image, (0.2,0.5),(0.2,0.5))\n    image = tf.image.random_flip_up_down(image)\n    image = tf.image.random_flip_left_right(image)\n    \n    return image\n    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Custom Data Generator","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Customer data generator is built which reads images from the disk and yields it.\nNeed knowledge on how \"Python Generators\" work to understand the below function.\nThe reason for building Custom Data Generator is that it gives you more flexibility[ Example : Augment only particular class of image/ augment only particular anatomical body images/ Center crop and resize only certain kind of images .. etc]\n\nCurrently all the images are augmented in the same way.. Will expore different options in the coming weeks ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def data_generator(image_dir,data,batch_size,num_batches,train = True): \n    '''\n    Generator which yields batch of images.\n    image_dir : Path to the images\n    data : Dataframe of the data\n    batch_size : Number of data points in a batch\n    num_batches : Number of batches\n    train : True/False \n    \n    '''\n        \n    while True:        \n        traversed_datapoints = 0\n        for batch in range(num_batches):            \n            i = 0\n            batch_data = tf.Variable(tf.zeros((batch_size,IMG_HEIGHT,IMG_WIDTH,3))) \n            batch_labels = tf.Variable(tf.zeros((batch_size,1),tf.int32)) \n            #print(batch_labels.dtype)\n            \n            while(i<batch_size):\n                \n                augmentation = False\n                image_path = os.path.join(image_dir, data['image_id'][i + traversed_datapoints] + '.jpg')\n                #print(image_path)\n                image = cv2.imread(image_path)              \n                image = image.astype(np.float32)/255.0\n                label = tf.reshape(int(data['target'][i + traversed_datapoints]) ,[1,] ) \n                #print(label.dtype)\n                if train:                    \n                    augmentation = True\n                    if augmentation:\n                        if np.random.randn(1)[0] > 0: \n                            #print(batch_labels[i].shape)\n                                                       \n\n                            image = data_augmentor(image)\n                            batch_data[i,:, :, :] = tf.Variable(image)                           \n                            \n                            batch_labels[i].assign(label)\n                            i = i + 1                       \n                        else:                            \n                             batch_data[i,:, :, :] = tf.Variable(image)\n                             batch_labels[i].assign(label)\n                             i = i + 1\n                    else:\n                       #print(batch_labels[i].shape)\n                       #print(int(data['target'][i + traversed_datapoints]))\n                       batch_data[i,:, :, :].assign(tf.Variable(image))\n                       batch_labels[i].assign(label)                      \n                       i = i + 1\n                else:\n                    batch_data[i,:, :, :].assign(tf.Variable(image)) \n                    i = i + 1                    \n            traversed_datapoints = batch_size*(batch+1)\n            \n            if data.shape[0] - traversed_datapoints < batch_size:\n                print(\"Modified batch size\")\n                batch_size = data.shape[0] - traversed_datapoints\n                print(batch_size)\n                                    \n            if train:\n                yield batch_data.numpy(), batch_labels.numpy()\n            else:\n                yield batch_data.numpy()\n\n        \n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\ndef fold_generator(fold):\n    '''\n    Function with takes in fold as an integer and returns the train and validation generators.\n    \n    '''\n       \n    train_data = df[df.fold != fold].reset_index(drop=True)\n    val_data = df[df.fold == fold].reset_index(drop=True)  \n    num_total = train_data.shape[0]\n    num_total_val = val_data.shape[0]\n    steps_per_epoch = math.ceil(num_total/batch_size)      \n    val_steps = int(num_total_val/batch_size)     \n    \n    train_data_generator = data_generator(train_dir,train_data,batch_size,steps_per_epoch,True)\n    val_data_generator = data_generator(train_dir,val_data,batch_size,steps_per_epoch,True)\n    \n\n    return train_data_generator, val_data_generator, train_data,val_data, steps_per_epoch, val_steps","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def scheduler(epoch):\n    '''\n    Simple Learning rate scheduler which exponentially decays the learning rate in every epoch\n    epoch : The current epoch number\n    \n    '''\n        \n    if epoch < 1:\n        return 0.0001\n    else:\n        return 0.00001 * tf.math.exp(0.1 * (10 - epoch))\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def focal_loss(y_true, y_pred):\n    gamma = 2.0\n    p_t    = (y_true*y_pred) + ((1-y_true)*(1-y_pred))        \n    scaling_factor = K.pow((1-p_t), gamma)\n    CE_loss = K.binary_crossentropy(y_true, y_pred)\n    focal_loss = scaling_factor*CE_loss\n    return focal_loss\n    \n    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## EfficientNetB1 Model","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Pretrained EfficientNet B1 Model is used with its top layer removed and adding a custom head","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\ndef MelnaomaNet(input_dim, base_model):\n    '''\n    Function with creates a model and return it\n    input_dim : Dimensions of the tensor input to the model\n    base_model : EfficientNet Model instance\n    \n    '''\n    \n    input_tensor = L.Input(input_dim)\n    curr_output  = base_model(input_tensor)\n    curr_output  = L.GlobalAveragePooling2D()(curr_output)\n    oputput      = L.Dense(1,activation='sigmoid')(curr_output)\n    model = Model(input_tensor, oputput)\n    opt = tf.keras.optimizers.Adam(learning_rate=0.00001)\n    model.compile(\n    optimizer = opt,\n    loss      = tf.keras.losses.binary_crossentropy,\n    metrics   = [tf.keras.metrics.AUC()]\n    )\n    return model\n\n\nwith strategy.scope():\n    dim = (512,512)\n    efnet = efn.EfficientNetB1(weights='imagenet', include_top = False, \n                       input_shape=(*dim, 3))\n    model = MelnaomaNet(input_dim=(*dim,3), base_model = efnet)\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Training","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Training the model for 10 Strafified KFolds","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"infer = False\n\nfor fold in range(n_splits):\n    '''\n    \n    Function which trains for each of the n_split fold and saves the model.\n    \n    '''\n    train_data_generator, val_data_generator, train_data, val_data, steps_per_epoch, val_steps = fold_generator(fold)\n\n    print(train_data['target'].value_counts())\n    print(val_data['target'].value_counts())\n    \n    checkpoint = ModelCheckpoint('../working/Model_fold_'+ str(fold)+'.h5', \n                             monitor='val_loss', \n                             verbose= 1,save_best_only=True, \n                             mode= 'min',save_weights_only=True,save_freq = 'epoch')\n    csv_logger = CSVLogger('../working/history.csv')\n\n\n    lr_schedule = tf.keras.callbacks.LearningRateScheduler(scheduler, verbose = 1)\n    early_stopping = tf.keras.callbacks.EarlyStopping(monitor = 'val_loss', patience = 1)\n    callbacks = [checkpoint, csv_logger,lr_schedule,early_stopping]\n    \n    if not infer:\n        train_history = model.fit_generator(\n            train_data_generator,\n            steps_per_epoch=steps_per_epoch,\n            validation_data=val_data_generator,\n            validation_steps = val_steps,\n            epochs=5,\n            callbacks = callbacks\n\n        )\n    else:\n        pass\n        ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Prediction","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"root =  '/kaggle/input/resize-jpg-siimisic-melanoma-classification/640x640/'\ntest_images  = os.path.join(root ,'test/')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_test = pd.DataFrame({\n    'image_name': os.listdir(test_images)\n})\n\ndf_test['image_name'] = df_test['image_name'].str.split('.').str[0]\nprint(df_test.shape)\ndf_test.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The final probability is calculated as the average of the probability of each of the 10 fold models","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# calling test generator\n\nsteps_per_epoch = math.ceil(df_test.shape[0]/batch_size)\n\n\nfinal_pred = np.zeros((df_test.shape[0],1))\nprint(final_pred.shape)\nfor fold in range(n_splits):\n    model.load_weights('../input/trainedfinal/Model_fold_'+str(fold)+'.h5')\n    test_generator = data_generator(test_images,df_test,batch_size,steps_per_epoch,False)\n    pred = model.predict(test_generator,steps=np.ceil(float(len(df_test)) / float(batch_size)), \n                                  verbose=1) \n    \n    print(pred.shape)\n    final_pred = final_pred + pred","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"final_pred = final_pred/n_splits","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(type(final_pred))\nprint(final_pred.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Submitting the csv file","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df_test['target'] = final_pred","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_test.to_csv('submission.csv', index=False)\ndf_test.head()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}