{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Introduction\nIn this project, I will show how one can finetune EfficientNet-B7 to detect Melanoma (a variety of skin cancer) from images. This problem is important because fast and accurate automated diagnosis can help reduce burden on doctors and let them focus on curing patients. I will use the Efficientnet Pytorch package to get the pretrained EfficientNet-B7 model and PyTorch XLA to train the model on TPU. At the end, we will run inference on the test set and test the model's predictions on some sample images.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Preparing the ground¶\nIn this section, we will prepare the ground to train and test the model by installing packages, setting hyperparameters, and loading the data.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":" ## Set up PyTorch XLA\nWe need to install a package called PyTorch XLA to take advantage of TPUs with PyTorch.\nTraining on TPUs can be significantly faster than on CPUs or GPUs. So we will use TPUs for this project.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install -q efficientnet","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!python pytorch-xla-env-setup.py --version nightly --apt-packages libomp5 libopenblas-dev","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Import necessary libraries\nNow, we import all the libraries we need.\ncolored, matplotlib, tqdm, and plotly for visualization.\nnumpy, pandas, torch, torchvision, albumentations, and efficientnet_pytorch for modelling.","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n%matplotlib inline\nfrom sklearn.preprocessing import LabelEncoder\n\n\nimport numpy as np\nimport pandas as pd \nimport os\nimport re\nimport cv2\nimport math\nimport time\nfrom tqdm import tqdm_notebook as tqdm\nfrom sklearn.model_selection import train_test_split\nfrom kaggle_datasets import KaggleDatasets\n\nimport tensorflow as tf\nimport tensorflow.keras.backend as K\nfrom tensorflow.keras.layers import Dense\nfrom tensorflow.keras.models import Model,Sequential\nfrom tensorflow.keras import optimizers\nimport efficientnet.tfkeras as efn\n\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom plotly.subplots import make_subplots\nimport plotly.express as px","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## TPU Configuration","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Tensorflow version \" + tf.__version__)\n\nAUTO = tf.data.experimental.AUTOTUNE\n\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy()\n     \nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)\n\nDATASET = '512x512-melanoma-tfrecords-70k-images'\nGCS_PATH = KaggleDatasets().get_gcs_path(DATASET)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Paths to train and test images\ntrain_img_path = '/kaggle/input/siim-isic-melanoma-classification/jpeg/train/'\ntest_img_path = '/kaggle/input/siim-isic-melanoma-classification/jpeg/test/'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Loading training and test dataset","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.DataFrame(pd.read_csv(\"/kaggle/input/siim-isic-melanoma-classification/train.csv\"))\ntest = pd.DataFrame(pd.read_csv(\"/kaggle/input/siim-isic-melanoma-classification/test.csv\"))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Exploratory Data Analysis","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train.shape, test.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### There are around 33k training images and about 10k testing images","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Identifying nan values in train data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train.isna().sum()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Training set : Sex, age and anatomy_site have missing values.\n      ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"test.isna().sum()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Test set : Anatomy_site have missing values.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Drop nan in train dataset","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train = train.dropna()\n\ntrain.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Drop nan in test dataset\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"test = test.dropna()\n\ntest.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Finding unique patient in the train and test dataset","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\ntrain['patient_id'].nunique(), test[\"patient_id\"].nunique()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\nThis means that out of 33,126 registered entries in the training set, only 2,056 are unique implying that some patients are diagnosed with multiple marks.\n\nSame goes for the test set where we have only 690 unique values out of collection of 10,982.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Distribution Observation of data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(train[\"target\"].value_counts())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Malignant VS Benign cases - OVERALL","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"malignant = len(train[train[\"target\"] == 1])\nbenign = len(train[train[\"target\"] == 0])\n\nlabels = [\"Malignant\", \"Benign\"] \nsize = [malignant, benign]\n\nplt.figure(figsize = (8, 8))\nplt.pie(size, labels = labels, shadow = True, startangle = 90, colors = [\"r\", \"g\"])\nplt.title(\"Malignant VS Benign Cases\")\nplt.legend()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Male VS Female Count\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_males = len(train[train[\"sex\"] == \"male\"])\ntrain_females  = len(train[train[\"sex\"] == \"female\"])\n\ntest_males = len(test[test[\"sex\"] == \"male\"])\ntest_females  = len(test[test[\"sex\"] == \"female\"])\nfig, ax = plt.subplots(1,2,figsize=(20,5))\nsns.countplot(x='sex',data=train,ax=ax[0])\nax[0].set_xlabel(\" \")\nax[0].set_title(\"Gender counts in train set\")\n\nprint(\"Number of males in training set = \", train_males)\nprint(\"Number of females in training set= \", train_females)\nsns.countplot(x='sex',data=test,ax=ax[1])\nax[1].set_xlabel(\" \")\nax[1].set_title(\"Gender counts in test set\")\nprint(\"Number of males in testing set = \", test_males)\nprint(\"Number of females in testing set= \", test_females)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"##  Malignant male cases VS female cases","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_malignant  = train[train[\"target\"] == 1]\ntrain_malignant_males = len(train_malignant[train_malignant[\"sex\"] == \"male\"])\ntrain_malignant_females  = len(train_malignant[train_malignant[\"sex\"] == \"female\"])\n\nlabels = [\"Malignant Male Cases\", \"Malignant Female Cases\"] \nsize = [train_malignant_males, train_malignant_females]\nexplode = [0.1, 0.0]\n\nplt.figure(figsize = (10, 10))\nplt.pie(size, labels = labels, explode = explode, shadow = True, startangle = 90, colors = [\"g\", \"b\"])\nplt.title(\"Malignant Male VS Female Cases\", fontsize = 18)\nplt.legend()\nprint(\"Malignant Male Cases = \", train_malignant_males)\nprint(\"Malignant Female Cases = \", train_malignant_females)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Benign male cases VS female cases","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_benign  = train[train[\"target\"] == 0]\n\ntrain_benign_males = len(train_benign[train_benign[\"sex\"] == \"male\"])\ntrain_benign_females  = len(train_benign[train_benign[\"sex\"] == \"female\"]) \n\nlabels = [\"Benign Male Cases\", \"Benign Female Cases\"] \nsize = [train_benign_males, train_benign_females]\nexplode = [0.1, 0.0]\n\nplt.figure(figsize = (10, 10))\nplt.pie(size, labels = labels, explode = explode, shadow = True, startangle = 90, colors = [\"g\", \"y\"])\nplt.title(\"Benign Male VS Benign Female Cases\", fontsize = 18)\nplt.legend()\nprint(\"Benign Male Cases = \", train_benign_males)\nprint(\"Benign Female Cases = \", train_benign_females)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Analysis of Cancer VS Sex","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"cancer_versus_sex = train.groupby([\"benign_malignant\",\"sex\"]).size()\nprint(cancer_versus_sex)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cancer_versus_sex = cancer_versus_sex.unstack(level = 1) / len(train) * 100\nprint(cancer_versus_sex)\nprint(type(cancer_versus_sex))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.set(style='whitegrid')\nsns.set_context(\"paper\", rc={\"font.size\":12,\"axes.titlesize\":20,\"axes.labelsize\":18})   \n\nplt.figure(figsize = (10, 6))\nsns.heatmap(cancer_versus_sex, annot=True, cmap=\"icefire\", cbar=True)\nplt.title(\"Cancer VS Sex Heatmap Analysis Normalized\", fontsize = 18)\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Analysing of Age Vs Cancer ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.set(style=\"whitegrid\")\nsns.set_context(\"paper\", rc={\"font_size\":12,\"axes.titlesize\":20,\"axes.labelsize\":18})   \nplt.figure(figsize = (10, 6))\nsns.boxplot(train[\"benign_malignant\"], train[\"age_approx\"], palette=\"icefire\")\nplt.title(\"Age VS Cancer Boxplot Analysis\")\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Inference :¶\n* The malignant cases belong to relatively higher age group.\n* Age might prove to be a contributing factor in deciding whether the case is malignant or benign.\n* From sex analysis we can hypothesize that sex might be a deciding factor as well","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# anatom_site_general_challenge","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"temp_train = train.anatom_site_general_challenge.value_counts().sort_values(ascending=False)\ntemp_test = test.anatom_site_general_challenge.value_counts().sort_values(ascending=False)\nprint(\"Anatom_Site valuecounts for train data\",temp_train)\nprint(\"Anatom_Site valuecounts for test data\",temp_test)\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nfig, ax = plt.subplots(1,2,figsize=(20,5))\nsns.barplot(x=temp_train.index.values, y=temp_train.values,ax=ax[0])\nax[0].set_xlabel(\" \")\nlabels = ax[0].get_xticklabels()\nax[0].set_xticklabels(labels, rotation=90)\nax[0].set_title(\"Image location in train set\")\n\nsns.barplot(x=temp_test.index.values, y=temp_test.values,ax=ax[1])\nax[1].set_xlabel(\" \")\nlabels = ax[1].get_xticklabels()\nax[1].set_xticklabels(labels, rotation=90)\nax[1].set_title(\"Image location in test set\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### It seems like majority of the cases are observed at the torso, and after that the extremities of the body (upper/lower) in both the training and testing set.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Age distribution in train and test dataset","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax =plt.subplots(1,2,figsize=(20,7))\nsns.countplot(x=\"age_approx\", data= train,ax=ax[0])\nax[0].set_xlabel(\"\")\nax[0].set_title(\"Age distribution in train set\")\nsns.countplot(x=\"age_approx\", data=test, ax=ax[1])\nax[0].set_xlabel(\"\")\nax[1].set_title(\"Age distribution in test set\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_ages_benign = train.loc[train[\"target\"] == 0, \"age_approx\"]\ntrain_ages_malignant = train.loc[train[\"target\"] == 1 , \"age_approx\"]\n\nplt.figure(figsize = (10, 8))\nsns.kdeplot(train_ages_benign, label = \"Benign\", shade = True, legend = True, cbar = True)\nsns.kdeplot(train_ages_malignant, label = \"Malignant\", shade = True, legend = True, cbar = True)\nplt.grid(True)\nplt.xlabel(\"Age Of The Patients\", fontsize = 18)\nplt.ylabel(\"Probability Density\", fontsize = 18)\nplt.grid(which = \"minor\", axis = \"both\")\nplt.title(\"Probabilistic Age Distribution In Training Set\", fontsize = 18)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Number of patients and samples","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print('No of samples:  ' + str(train.image_name.nunique()))\nprint('No of patients: ' + str(train.patient_id.nunique()))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### No of samples taken from patients frequency\nObserving the number of patients and no of total samples,I came to the follwoing insights.\n\nAll the patients gave at least 2 samples.\nMaximum no of sample taken from a single patient is 115.\nOn an average each patient gave 16 samples\nMedian of samples of image per patient is 12\nMode of samples of image per patient is 2","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"image_freq_per_patient = train.groupby(['patient_id']).count()['image_name']\nplt.hist(image_freq_per_patient.tolist(), bins = image_freq_per_patient.nunique())\nplt.xlabel('No of samples per patient')\nplt.ylabel('No of patients')\nplt.show()\nprint('Minimum no of sample taken from  single patient', image_freq_per_patient.min())\nprint('Maximum no of sample taken from  single patient', image_freq_per_patient.max())\nprint('There are ',int( image_freq_per_patient.mean()), ' samples taken from each patients on average')\nprint('Median of no. of samples taken from  single patient', int(image_freq_per_patient.median()))\nprint('Mode of no. of samples taken from  single patient', int(image_freq_per_patient.mode()))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Here we observe that among the unique patients providing samples,\n\nMelanoma is more prevalant in Women\nAmong the Male patients, almost 24% are at malignant stage\nOn the other hand, among Femele patients, about 17% are at malignant stage","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train.groupby(['benign_malignant', 'sex']).nunique()['patient_id']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"category_sex = train.groupby(['sex', 'benign_malignant']).nunique()['patient_id'].tolist()\n\nlabels = ['Benign', 'Malignant']\nbenign_data = category_sex[0:2]\nmaglignant_data = category_sex[2:4]\nx = np.arange(len(labels))  # the label locations\nwidth = 0.35  # the width of the bars\n\nfig, ax = plt.subplots()\nrects1 = ax.bar(x - width/2, benign_data, width, label='Male')\nrects2 = ax.bar(x + width/2, maglignant_data, width, label='Female')\nax.set_ylabel('No of patients')\nax.set_title('Patient Count by Benign and Malignant with Sex')\nax.set_xticks(x)\nax.set_xticklabels(labels)\nax.legend()\n\ndef autolabel(rects):\n    \"\"\"Attach a text label above each bar in *rects*, displaying its height.\"\"\"\n    for rect in rects:\n        height = rect.get_height()\n        ax.annotate('{}'.format(height),\n                    xy=(rect.get_x() + rect.get_width() / 2, height),\n                    xytext=(0, 3),  # 3 points vertical offset\n                    textcoords=\"offset points\",\n                    ha='center', va='bottom')\n        \nautolabel(rects1)\nautolabel(rects2)\n\nfig.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### One Hot Encoding¶\nTransforming all categorical features un numerical.\n\nNote1: sex, anatomy, diagnosis need to be encoded.\n\nNote2: benign_malignant column will be dropped, as the information is already in the target column.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# === TRAIN ===\nto_encode = ['sex', 'anatom_site_general_challenge', 'diagnosis']\nencoded_all = []\n\nlabel_encoder = LabelEncoder()\n\nfor column in to_encode:\n    encoded = label_encoder.fit_transform(train[column])\n    encoded_all.append(encoded)\n    \ntrain['sex'] = encoded_all[0]\ntrain['anatom_site_general_challenge'] = encoded_all[1]\ntrain['diagnosis'] = encoded_all[2]\n\nif 'benign_malignant' in train.columns : train.drop(['benign_malignant'], axis=1, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# === TEST ===\nto_encode = ['sex', 'anatom_site_general_challenge']\nencoded_all = []\n\nlabel_encoder = LabelEncoder()\n\nfor column in to_encode:\n    encoded = label_encoder.fit_transform(test[column])\n    encoded_all.append(encoded)\n    \ntest['sex'] = encoded_all[0]\ntest['anatom_site_general_challenge'] = encoded_all[1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Loading the train and test images","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"image_names = train[\"image_name\"].values\nimage_names = image_names + \".jpg\"\nimage_names","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### We generate 4 random samples from the training data set. These 4 samples are taken from the aforementioned array of names.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"random_images = [np.random.choice(image_names) for i in range(4)] # Generates a random sample from a given 1-D array\nrandom_images","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Sample Images","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"#### Earlier we extracted paths of all directories. So, we will access these images from there.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_dir = \"/kaggle/input/siim-isic-melanoma-classification/jpeg/train/\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize = (12, 8))\nfor i in range(4) : \n    plt.subplot(2, 2, i + 1) \n    image = cv2.imread(os.path.join(train_dir, random_images[i]))\n    # cv2 reads images in BGR format. Hence we convert it to RGB\n    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n    plt.imshow(image, cmap = \"gray\")\n    plt.grid(True)\n# Automatically adjust subplot parameters to give specified padding.\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Analysis of color distribution","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Benign Cases :\n\nSince we have a lot of images here, hence we randomly sample only a thousand of them.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Benign Cases\nbenign = train[train[\"target\"] == 0] \nimage_names = benign[\"image_name\"].values\nimage_names = image_names + \".jpg\"\nbenign_image_list = [np.random.choice(image_names) for i in tqdm(range(1000))]\n\nred = []\ngreen = [] \nblue = []\n\nfor image_name in tqdm(benign_image_list) : \n    image = cv2.imread(os.path.join(train_dir, image_name))\n    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n    \n    mean_red = np.mean(image[:,:,0])\n    mean_green = np.mean(image[:,:,1])\n    mean_blue = np.mean(image[:,:,2])\n    \n    red.append(mean_red)\n    green.append(mean_green)\n    blue.append(mean_blue)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Define hyperparameters and paths","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"SEED = 42\nBATCH_SIZE = 32 * strategy.num_replicas_in_sync\nSIZE = [512,512]\nLR = 0.00004\nEPOCHS = 12\nWARMUP = 5\nWEIGHT_DECAY = 0\nLABEL_SMOOTHING = 0.05\nTTA = 4","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def seed_everything(SEED):\n    np.random.seed(SEED)\n    tf.random.set_seed(SEED)\n\nseed_everything(SEED)\ntrain_filenames = tf.io.gfile.glob(GCS_PATH + '/train*.tfrec')\ntest_filenames = tf.io.gfile.glob(GCS_PATH + '/test*.tfrec')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_filenames,valid_filenames = train_test_split(train_filenames,test_size = 0.2,random_state = SEED)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def decode_image(image):\n    image = tf.image.decode_jpeg(image, channels=3) \n    image = tf.cast(image, tf.float32)/255.0\n    image = tf.reshape(image, [*SIZE, 3])\n    return image\n\ndef data_augment(image, label=None, seed=SEED):\n    image = tf.image.rot90(image,k=np.random.randint(4))\n    image = tf.image.random_flip_left_right(image, seed=seed)\n    image = tf.image.random_flip_up_down(image, seed=seed)\n    if label is None:\n        return image\n    else:\n        return image, label\n\ndef read_labeled_tfrecord(example):\n    LABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string),\n        \"target\": tf.io.FixedLenFeature([], tf.int64),  }\n    example = tf.io.parse_single_example(example, LABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    label = tf.cast(example['target'], tf.int32)\n    return image, label \n\ndef read_unlabeled_tfrecord(example):\n    UNLABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string),\n        \"image_name\": tf.io.FixedLenFeature([], tf.string), }\n    example = tf.io.parse_single_example(example, UNLABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    image_name = example['image_name']\n    return image, image_name\n\ndef load_dataset(filenames, labeled=True, ordered=False):\n    ignore_order = tf.data.Options()\n    if not ordered:\n        ignore_order.experimental_deterministic = False\n\n    dataset = (tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTO) \n              .with_options(ignore_order)\n              .map(read_labeled_tfrecord if labeled else read_unlabeled_tfrecord, num_parallel_calls=AUTO))\n            \n    return dataset\n\ndef count_data_items(filenames):\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n    return np.sum(n)\n\ndef plot_transform(num_images):\n    plt.figure(figsize=(30,10))\n    x = load_dataset(train_filenames, labeled=False)\n    image,_ = iter(x).next()\n    for i in range(1,num_images+1):\n        plt.subplot(1,num_images+1,i)\n        plt.axis('off')\n        image = data_augment(image=image)\n        plt.imshow(image)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_transform(7)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_dataset = (load_dataset(train_filenames, labeled=True)\n    .map(data_augment, num_parallel_calls=AUTO)\n    .shuffle(SEED)\n    .batch(BATCH_SIZE,drop_remainder=True)\n    .repeat()\n    .prefetch(AUTO))\n\nvalid_dataset = (load_dataset(valid_filenames, labeled=True)\n    .batch(BATCH_SIZE)\n    .cache()\n    .prefetch(AUTO))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## EfficientNetB7","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n\n    model = tf.keras.Sequential([\n        efn.EfficientNetB7(input_shape=(*SIZE, 3),weights='imagenet',pooling='avg',include_top=False),\n        Dense(1, activation='sigmoid')\n    ])\n    \n    model.compile(\n        optimizer='adam',\n        loss = tf.keras.losses.BinaryCrossentropy(label_smoothing = LABEL_SMOOTHING),\n        metrics=['accuracy',tf.keras.metrics.AUC(name='auc')])\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Learning Rate Schedules","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_cosine_schedule_with_warmup(lr,num_warmup_steps, num_training_steps, num_cycles=0.5):\n    def lrfn(epoch):\n        if epoch < num_warmup_steps:\n            return (float(epoch) / float(max(1, num_warmup_steps))) * lr\n        progress = float(epoch - num_warmup_steps) / float(max(1, num_training_steps - num_warmup_steps))\n        return max(0.0, 0.5 * (1.0 + math.cos(math.pi * float(num_cycles) * 2.0 * progress))) * lr\n\n    return tf.keras.callbacks.LearningRateScheduler(lrfn, verbose=True)\n\nlr_schedule= get_cosine_schedule_with_warmup(lr=LR,num_warmup_steps=WARMUP,num_training_steps=EPOCHS)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def train():\n        STEPS_PER_EPOCH = count_data_items(train_filenames) // BATCH_SIZE\n        history = model.fit(\n            train_dataset, \n            epochs=EPOCHS, \n            callbacks=[lr_schedule],\n            steps_per_epoch=STEPS_PER_EPOCH,\n            validation_data=valid_dataset)\n\n        string = 'Train acc:{:.4f} Train loss:{:.4f} AUC: {:.4f}, Val acc:{:.4f} Val loss:{:.4f} Val AUC: {:.4f}'.format( \\\n            model.history.history['accuracy'][-1],model.history.history['loss'][-1],\\\n            model.history.history['auc'][-1],\\\n            model.history.history['val_accuracy'][-1],model.history.history['val_loss'][-1],\\\n            model.history.history['val_auc'][-1])\n\n        return string","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Prediction Based on Test Time Augmentation (TTA)\n* Data augmentation technique for the test dataset, where multiple augmentaed copies of images in dataset is created with zoom, flip and shifts\n* The artificially expanded training dataset can result in a more skillful model, as often the performance of deep learning models continues to scale in concert with the size of the training dataset\n* The model makes prediction for each and then ensemble of the predictions are returned","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"num_test_images = count_data_items(test_filenames)\nsubmission_df = pd.read_csv('/kaggle/input/siim-isic-melanoma-classification/sample_submission.csv')\nfor i in range(TTA):\n    test_dataset = (load_dataset(test_filenames, labeled=False,ordered=True)\n    .map(data_augment, num_parallel_calls=AUTO)  \n    .batch(BATCH_SIZE))\n    test_dataset_images = test_dataset.map(lambda image, image_name: image)\n    test_dataset_image_name = test_dataset.map(lambda image, image_name: image_name).unbatch()\n    test_ids = next(iter(test_dataset_image_name.batch(num_test_images))).numpy().astype('U')\n    test_pred = model.predict(test_dataset_images, verbose=1) \n    pred_df = pd.DataFrame({'image_name': test_ids, 'target': np.concatenate(test_pred)})\n    temp = submission_df.copy()   \n    del temp['target']  \n    submission_df['target'] += temp.merge(pred_df,on=\"image_name\")['target']/TTA","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission_df.to_csv('submission.csv', index=False)\npd.Series(np.round(submission_df['target'].values)).value_counts() ","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}