{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"### This is just data visualization notebook. I will gradually update and publish this notebook over time. Your kind suggestion is highly appreciated.  If you find this insightful, please upvote it. Thanks. ","metadata":{}},{"cell_type":"markdown","source":"## Data Format\nThe images are provided in DICOM format. This can be accessed using commonly-available libraries like **`pydicom`**, and contains both image and metadata. It is a commonly used medical imaging data format.\n\nImages are also provided in ** `JPEG`** and** `TFRecord`** format (in the jpeg and tfrecords directories, respectively). Images in **`TFRecord`** format have been resized to a uniform ** `1024x1024`**.\n\nMetadata is also provided outside of the DICOM format, in CSV files. See the Columns section for a description.\n\n## What to predict. \nWe have to predict a binary target for each image. The model model should predict the probability (floating point) between 0.0 and 1.0 that the lesion in the image is malignant (the target). In the training data, `train.csv`, the **value 0 denotes benign, and 1 indicates malignant.**\n\n## Data Set Files\nThe dataset consists of images in :\n* DICOM format in the directories `train` and `test`\n* JPEG format in JPEG directory\n* TFRecord format in tfrecords directory\n\nAdditionally, there is a metadata comprising of train, test and submission file in CSV format.\nSo the whole dataset looks like the following\n* **jpeg(dir)**\n    * test --> all the jpg images in testset\n    * train --> all the jpg images in training  set\n* ** test(dir)** --> all the test images in DICOM format\n* **tfrecords(dir)**  --> dataset converted into TFRecords\n* **train(dir)**  -->all the training images in DICOM format\n* **train.csv** --> the training set metadata\n* **test.csv**  -->the test set metadata\n* sample_submission.csv --> a sample submission file in the correct format\n","metadata":{}},{"cell_type":"markdown","source":"## Evaluation Metrics\nThe metric in evaluation for this competition is AUC, that for \"Area under the ROC Curve.\"\nSo lets know about ROC Curve. \n### ROC curve\nAn **ROC curve (receiver operating characteristic curve)** is a graph showing the performance of a classification model at all classification thresholds. Let us take the following confusion matrix. ![confusion_matrix](https://miro.medium.com/max/356/1*g5zpskPaxO8uSl0OWT4NTQ.png)\nThis curve plots two parameters:\n* True Positive Rate\n* False Positive Rate\n**True Positive Rate (TPR)** is a synonym for recall and is therefore defined as follows:\n$$TPR = \\frac{TP}{(TP + FN)}$$\n\n**False Positive Rate (FPR)** is defined as follows:\n$$FPR =\\frac{ FP}{  (FP + TN)}$$\n\n**An ROC curve plots TPR vs. FPR at different classification thresholds**. Lowering the classification threshold classifies more items as positive, thus increasing both False Positives and True Positives. The following figure shows a typical ROC curve.\n![](https://imgur.com/N3UOcBF.png)\n\n## AUC: Area Under the ROC Curve\n**AUC** stands for **\"Area under the ROC Curve.\"**  That is, AUC measures the entire two-dimensional area underneath the entire ROC curve (think integral calculus) from (0,0) to (1,1).\n![auc](https://imgur.com/YE18DBT.jpg =280x)\nAUC provides an aggregate measure of performance across all possible classification thresholds. One way of interpreting AUC is as the probability that the model ranks a random positive example more highly than a random negative example.\n\n\nSource: [Classification: ROC Curve and AUC](https://developers.google.com/machine-learning/crash-course/classification/roc-and-auc)","metadata":{}},{"cell_type":"markdown","source":"# 1. Loading Libraries","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true}},{"cell_type":"code","source":"!pip install -q efficientnet\n!pip install -q pyyaml h5py\n\n#basic libraries\nimport os, re, math\nimport numpy as np\nimport pandas as pd\n\n#plot libraries\nimport matplotlib.pyplot as plt\nimport plotly.express as px\n\n#utilities library\nfrom sklearn import metrics\nfrom sklearn.model_selection import train_test_split\n\n#background library for learning \nimport tensorflow as tf\nimport tensorflow.keras.layers as Layers\n\nfrom kaggle_datasets import KaggleDatasets\n\nimport efficientnet.tfkeras as efn\n\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df = pd.read_csv('/kaggle/input/siim-isic-melanoma-classification/train.csv')\ntest_df = pd.read_csv('/kaggle/input/siim-isic-melanoma-classification/test.csv')\nsub = pd.read_csv('/kaggle/input/siim-isic-melanoma-classification/sample_submission.csv')","metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2. Metadata Description\n### Columns of `train.csv`\n* image_name - unique identifier, points to filename of related DICOM image\n* patient_id - unique patient identifier\n* sex - the sex of the patient (when unknown, will be blank)\n* age_approx - approximate patient age at time of imaging\n* anatom_site_general_challenge - location of imaged site\n* diagnosis - detailed diagnosis information (train only)\n* benign_malignant - indicator of malignancy of imaged lesion\n* target - binarized version of the target variable","metadata":{}},{"cell_type":"code","source":"print('No of samples:  ' + str(train_df.image_name.nunique()))\nprint('No of patients: ' + str(train_df.patient_id.nunique()))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### No of samples taken from patients frequency\nObserving the number of patients and no of total samples,I came to the follwoing insights. \n\n* All the patients gave at least 2 samples.\n* Maximum no of sample taken from a single patient is 115.\n* On an average each patient gave 16 samples\n* Median of samples of image per patient is 12\n* Mode of samples of image per patient is 2","metadata":{}},{"cell_type":"code","source":"image_freq_per_patient = train_df.groupby(['patient_id']).count()['image_name']\nplt.hist(image_freq_per_patient.tolist(), bins = image_freq_per_patient.nunique())\nplt.xlabel('No of samples per patient')\nplt.ylabel('No of patients')\nplt.show()\nprint('Minimum no of sample taken from  single patient', image_freq_per_patient.min())\nprint('Maximum no of sample taken from  single patient', image_freq_per_patient.max())\nprint('There are ',int( image_freq_per_patient.mean()), ' samples taken from each patients on average')\nprint('Median of no. of samples taken from  single patient', int(image_freq_per_patient.median()))\nprint('Mode of no. of samples taken from  single patient', int(image_freq_per_patient.mode()))\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sex_count = train_df.groupby(['sex']).count()['image_name']\nfig = plt.figure()\nax = fig.add_axes([0,0,1,1])\nax.bar(['Female', 'Male'], sex_count)\nplt.ylabel('count')\nplt.show()\nsex_count","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"category_sex = train_df.groupby(['sex', 'benign_malignant']).nunique()['patient_id'].tolist()\n\nlabels = ['Benign', 'Malignant']\nbenign_data = category_sex[0:2]\nmaglignant_data = category_sex[2:4]\nx = np.arange(len(labels))  # the label locations\nwidth = 0.35  # the width of the bars\n\nfig, ax = plt.subplots()\nrects1 = ax.bar(x - width/2, benign_data, width, label='Male')\nrects2 = ax.bar(x + width/2, maglignant_data, width, label='Female')\nax.set_ylabel('No of patients')\nax.set_title('Patient Count by Benign and Malignant with Sex')\nax.set_xticks(x)\nax.set_xticklabels(labels)\nax.legend()\n\ndef autolabel(rects):\n    \"\"\"Attach a text label above each bar in *rects*, displaying its height.\"\"\"\n    for rect in rects:\n        height = rect.get_height()\n        ax.annotate('{}'.format(height),\n                    xy=(rect.get_x() + rect.get_width() / 2, height),\n                    xytext=(0, 3),  # 3 points vertical offset\n                    textcoords=\"offset points\",\n                    ha='center', va='bottom')\n        \nautolabel(rects1)\nautolabel(rects2)\n\nfig.tight_layout()\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here we observe that among the unique patients providing samples, \n* **Melanoma is more prevalant in Women** \n* **Among the Male patients, almost 24% are at malignant stage**\n* **On the other hand, among Femele patients, about 17% are at malignant stage**","metadata":{}},{"cell_type":"code","source":"train_df.groupby(['benign_malignant', 'sex']).nunique()['patient_id']","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure()\ntrain_df.groupby(['benign_malignant']).mean()['age_approx'].plot.bar(x = 'Diagnosis Type', y = 'Average age', rot = 0)\nplt.title('Benign/Malignant vs Average Age')\nplt.xlabel('Diagnosis Outcome')\nplt.ylabel('Average Approx. Age')\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here we see that comparatively malignant patients are of higher age and their average age is almost 57 years whereas average age of the benign patients is close to 50. ","metadata":{}},{"cell_type":"code","source":"site_vs_diagnosis = train_df.groupby(['anatom_site_general_challenge', 'benign_malignant']).count()['patient_id'].tolist()\nlabels = ['head/neck', 'lower extremity', 'oral/genital','palms/soles', 'torso', 'upper extremity']\nbenign_data = site_vs_diagnosis[0:12:2]\nmaglignant_data = site_vs_diagnosis[1:12:2]\nx = np.arange(len(labels))  # the label locations\nwidth = 0.35  # the width of the bars\n\nfig, ax = plt.subplots(figsize = (10,6))\nrects1 = ax.bar(x - width/2, benign_data, width, label='Benign')\nrects2 = ax.bar(x + width/2, maglignant_data, width, label='Malignant')\nax.set_ylabel('No of samples')\nax.set_xlabel('Anatomical Sites')\nax.set_title('Patient Count by Benign and Malignant with Anatomical Location')\nax.set_xticks(x)\nax.set_xticklabels(labels)\nax.legend()\n\ndef autolabel(rects):\n    \"\"\"Attach a text label above each bar in *rects*, displaying its height.\"\"\"\n    for rect in rects:\n        height = rect.get_height()\n        ax.annotate('{}'.format(height),\n                    xy=(rect.get_x() + rect.get_width() / 2, height),\n                    xytext=(0, 3),  # 3 points vertical offset\n                    textcoords=\"offset points\",\n                    ha='center', va='bottom')\n        \nautolabel(rects1)\nautolabel(rects2)\n\nfig.tight_layout()\nplt.show()\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**So the above graph says that oral/genital and palms/soles are least likely to develop Melanoma disease whereas most of the time torso develops melanoma**\n","metadata":{}},{"cell_type":"markdown","source":"# 4. TPU Setup Code","metadata":{}},{"cell_type":"code","source":"try:\n    # TPU detection. No parameters necessary if TPU_NAME environment variable is\n    # set: this is always the case on Kaggle.\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    # Default distribution strategy in Tensorflow. Works on CPU and single GPU.\n    strategy = tf.distribute.get_strategy()\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 5. Setting up Running Configuration ","metadata":{}},{"cell_type":"code","source":"# For tf.dataset Tensorflow tf. data AUTOTUNE. ... prefetch transformation, \n# which can be used to decouple the time when data is produced from the time when data is consumed. \n# In particular, the transformation uses a background thread and an internal buffer to prefetch \n# elements from the input dataset ahead of the time they are requested.\nAUTO = tf.data.experimental.AUTOTUNE\n\n# Get data access to the dataset for TPUs\nGCS_PATH = KaggleDatasets().get_gcs_path('siim-isic-melanoma-classification')\n\n# Running Configuration \nEPOCHS = 15\nBATCH_SIZE = 8 * strategy.num_replicas_in_sync\nIMAGE_SIZE = [1024, 1024]\n\n# Listing the filenames in TFRecords fomat\nTRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/tfrecords/train*.tfrec')\nTEST_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/tfrecords/test*.tfrec')\nCLASSES = [0,1]\nprint('Training filenames\\n', TRAINING_FILENAMES)\nprint('Test file names\\n', TEST_FILENAMES)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 6. Training Validation Split","metadata":{}},{"cell_type":"code","source":"VALIDATION_FILENAMES =list(pd.Series(TRAINING_FILENAMES)[[0,1,2]])\nTRAINING_FILENAMES = list(pd.Series(TRAINING_FILENAMES)[[3,4,5,6,7,8,9,10,11,12,13,14,15]])\nprint(TRAINING_FILENAMES)\nprint(VALIDATION_FILENAMES)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import seaborn as sns\nsns.set(style=\"darkgrid\")\nsns.countplot(train_df['benign_malignant'])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 6. Helper Functions","metadata":{}},{"cell_type":"code","source":"def decode_image(image_data):\n    image = tf.image.decode_jpeg(image_data, channels=3)\n    image = tf.cast(image, tf.float32) / 255.0  # convert image to floats in [0, 1] range\n    image = tf.reshape(image, [*IMAGE_SIZE, 3]) # explicit size needed for TPU\n    return image\n\ndef read_labeled_tfrecord(example):\n    LABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        #\"class\": tf.io.FixedLenFeature([], tf.int64),  # shape [] means single element\n        \"target\": tf.io.FixedLenFeature([], tf.int64),  # shape [] means single element\n    }\n    example = tf.io.parse_single_example(example, LABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    #label = tf.cast(example['class'], tf.int32)\n    label = tf.cast(example['target'], tf.int32)\n    return image, label # returns a dataset of (image, label) pairs\n\ndef read_unlabeled_tfrecord(example):\n    UNLABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"image_name\": tf.io.FixedLenFeature([], tf.string),  # shape [] means single element\n        # class is missing, this competitions's challenge is to predict flower classes for the test dataset\n    }\n    example = tf.io.parse_single_example(example, UNLABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    idnum = example['image_name']\n    return image, idnum # returns a dataset of image(s)\n\ndef load_dataset(filenames, labeled=True, ordered=False):\n    # Read from TFRecords. For optimal performance, reading from multiple files at once and\n    # disregarding data order. Order does not matter since we will be shuffling the data anyway.\n\n    ignore_order = tf.data.Options()\n    if not ordered:\n        ignore_order.experimental_deterministic = False # disable order, increase speed\n\n    dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTO) # automatically interleaves reads from multiple files\n    dataset = dataset.with_options(ignore_order) # uses data as soon as it streams in, rather than in its original order\n    dataset = dataset.map(read_labeled_tfrecord if labeled else read_unlabeled_tfrecord, num_parallel_calls=AUTO)\n    # returns a dataset of (image, label) pairs if labeled=True or (image, id) pairs if labeled=False\n    return dataset\n\n\ndef get_training_dataset():\n    dataset = load_dataset(TRAINING_FILENAMES, labeled=True)\n    dataset = dataset.map(data_augment, num_parallel_calls=AUTO)\n    dataset = dataset.repeat() # the training dataset must repeat for several epochs\n    dataset = dataset.shuffle(2048)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\ndef get_validation_dataset(ordered=False):\n    dataset = load_dataset(VALIDATION_FILENAMES, labeled=True, ordered=ordered)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.cache()\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\ndef get_test_dataset(ordered=False):\n    dataset = load_dataset(TEST_FILENAMES, labeled=False, ordered=ordered)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\ndef count_data_items(filenames):\n    # the number of data items is written in the name of the .tfrec files, i.e. flowers00-230.tfrec = 230 data items\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n    return np.sum(n)\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 7. Data Augmentation ","metadata":{}},{"cell_type":"code","source":"def data_augment(image, label):\n    # data augmentation. Thanks to the dataset.prefetch(AUTO) statement in the next function (above),\n    # this happens essentially for free on TPU. Data pipeline code is executed on the \"CPU\" part\n    # of the TPU while the TPU itself is computing gradients.\n    image = tf.image.random_flip_left_right(image)\n#     image = tf.image.random_flip_up_down(image)\n    #image = tf.image.random_saturation(image, 0, 2)\n    return image, label   ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 8. Data Sumamry","metadata":{}},{"cell_type":"code","source":"NUM_TRAINING_IMAGES = count_data_items(TRAINING_FILENAMES)\nNUM_TEST_IMAGES = count_data_items(TEST_FILENAMES)\nNUM_VALID_IMAGES = count_data_items(VALIDATION_FILENAMES)\nSTEPS_PER_EPOCH = NUM_TRAINING_IMAGES // BATCH_SIZE\n\nprint('Dataset Details:\\n{} training images,  \\n{} validation images \\n{} unlabeled test images'.format(NUM_TRAINING_IMAGES, NUM_VALID_IMAGES, NUM_TEST_IMAGES))\n\n\ndf = pd.DataFrame({'data':['NUM_TRAINING_IMAGES', 'NUM_TEST_IMAGES'],\n                   'No of Samples':[NUM_TRAINING_IMAGES, NUM_TEST_IMAGES]})\nplt.figure()\nx = df.plot.bar(x='data', y='No of Samples', rot=0)\nplt.ylabel('No of Samples')\nplt.title('No of Training and Test Images')\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 9. Learning Rate Scheduler","metadata":{}},{"cell_type":"code","source":"def build_lrfn(lr_start=0.000002, lr_max=0.00010, \n               lr_min=0, lr_rampup_epochs=8, \n               lr_sustain_epochs=1, lr_exp_decay=.8):\n\n    def lrfn(epoch):\n        if epoch < lr_rampup_epochs:\n            lr = (lr_max - lr_start) / lr_rampup_epochs * epoch + lr_start\n        elif epoch < lr_rampup_epochs + lr_sustain_epochs:\n            lr = lr_max\n        else:\n            lr = (lr_max - lr_min) *\\\n                 lr_exp_decay**(epoch - lr_rampup_epochs\\\n                                - lr_sustain_epochs) + lr_min\n        return lr\n    return lrfn\n\nlrfn = build_lrfn()\nlr_schedule = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose=True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_checkpoint_callback_efnB0 = tf.keras.callbacks.ModelCheckpoint(\n    filepath='model_efnB0_best_val_acc.hdf5',\n    save_weights_only=True,\n    monitor='val_accuracy',\n    mode='max',\n    save_best_only=True)\n\nmodel_checkpoint_callback_efnB1 = tf.keras.callbacks.ModelCheckpoint(\n    filepath='model_efnB1_best_val_acc.hdf5',\n    save_weights_only=True,\n    monitor='val_accuracy',\n    mode='max',\n    save_best_only=True)\n\nmodel_checkpoint_callback_efnB3 = tf.keras.callbacks.ModelCheckpoint(\n    filepath='model_efnB3_best_val_auc.hdf5',\n    save_weights_only=True,\n    monitor='val_auc',\n    mode='max',\n    save_best_only=True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 10. Training Models","metadata":{}},{"cell_type":"markdown","source":"## EfficentNetB5 ","metadata":{}},{"cell_type":"code","source":"# with strategy.scope():\n#     model0 = tf.keras.Sequential([\n#         efn.EfficientNetB5(\n#             input_shape=(*IMAGE_SIZE, 3),\n#             #weights='noisy-student',\n#             weights='imagenet',\n#             include_top=False\n#         ),\n#         Layers.GlobalAveragePooling2D(),\n#         Layers.Dense(1024, activation = 'relu'), \n#         Layers.Dropout(0.3), \n#         Layers.Dense(512, activation= 'relu'), \n#         Layers.Dropout(0.2), \n#         Layers.Dense(256, activation='relu'), \n#         Layers.Dropout(0.2), \n#         Layers.Dense(128, activation='relu'), \n#         Layers.Dropout(0.1), \n#         Layers.Dense(1, activation='sigmoid')\n#     ])\n    \n# model0.compile(\n#     optimizer='adam',\n#     loss = tf.keras.losses.BinaryCrossentropy(label_smoothing = 0.1),\n#     metrics=['binary_crossentropy']\n# )\n# model0.summary()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## EfficentNetB7","metadata":{}},{"cell_type":"code","source":"# with strategy.scope():\n#     model1 = tf.keras.Sequential([\n#         efn.EfficientNetB7(\n#             input_shape=(*IMAGE_SIZE, 3),\n#             weights='imagenet',\n#             include_top=False\n#         ),\n#         Layers.GlobalAveragePooling2D(),\n#         Layers.Dense(1, activation='sigmoid')\n#     ])\n#     model1.compile(\n#         optimizer='adam',\n#         loss = 'binary_crossentropy',\n#         metrics=['accuracy']\n#     )\n#     model1.summary()\n    ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## EfficentNetB0","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    model_efn_b0 = tf.keras.Sequential([\n        efn.EfficientNetB0(\n            input_shape=(*IMAGE_SIZE, 3),\n            weights='imagenet',\n            include_top=False\n        ),\n        Layers.GlobalAveragePooling2D(),\n        Layers.Dense(1, activation='sigmoid')\n    ])\n    model_efn_b0.compile(\n        optimizer='adam',\n        loss = 'binary_crossentropy',\n        metrics=['accuracy']\n    )\n    model_efn_b0.summary()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# EfficentNetB1","metadata":{}},{"cell_type":"code","source":"# with strategy.scope():\n#     model_efn_b1 = tf.keras.Sequential([\n#         efn.EfficientNetB1(\n#             input_shape=(*IMAGE_SIZE, 3),\n#             weights='imagenet',\n#             include_top=False\n#         ),\n#         Layers.GlobalAveragePooling2D(),\n#         Layers.Dense(1, activation='sigmoid')\n#     ])\n#     model_efn_b1.compile(\n#         optimizer='adam',\n#         loss = 'binary_crossentropy',\n#         metrics=['accuracy']\n#     )\n#     model_efn_b1.summary()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# EfficentNetB3","metadata":{}},{"cell_type":"code","source":"# with strategy.scope():\n#     model_efn_b3 = tf.keras.Sequential([\n#         efn.EfficientNetB3(\n#             input_shape=(*IMAGE_SIZE, 3),\n#             #weights='imagenet',\n#             weights='imagenet',\n#             include_top=False\n#         ),\n#         Layers.GlobalAveragePooling2D(),\n#         Layers.Dense(512, activation='relu'),\n#         Layers.Dense(128, activation='relu'),\n#         Layers.Dense(1, activation='sigmoid')\n#     ])\n    \n# model_efn_b3.compile(\n#     optimizer='adam',\n#     loss = 'binary_crossentropy',\n#     metrics=['accuracy', tf.keras.metrics.AUC()]\n# )\n# model_efn_b3.summary()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## DenseNet201","metadata":{}},{"cell_type":"code","source":"# from tensorflow.keras.applications import DenseNet201\n# with strategy.scope():\n#     dnet201 = DenseNet201(\n#         input_shape=(*IMAGE_SIZE, 3),\n#         weights='imagenet',\n#         include_top=False\n#     )\n#     dnet201.trainable = True\n\n#     model3 = tf.keras.Sequential([\n#         dnet201,\n#         Layers.GlobalAveragePooling2D(),\n#         Layers.Dense(1, activation='sigmoid')\n#     ])\n#     model3.compile(\n#         optimizer='adam',\n#         loss = 'binary_crossentropy',\n#         metrics=['accuracy']\n#     )\n# model3.summary()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# history = model1.fit(\n#     get_training_dataset(), \n#     epochs=2, \n#     callbacks=[lr_schedule],\n#     steps_per_epoch=NUM_TRAINING_IMAGES // BATCH_SIZE\n#     #validation_data=valid_dataset\n# )","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# tf.tpu.experimental.initialize_tpu_system(tpu)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\nmodel_efn_b0.load_weights('/kaggle/input/melanoma-pretrained/model_efnB0_best_val_acc(just_5_epoch_895).hdf5')\n# history_efn_b0 = model_efn_b0.fit(\n#     get_training_dataset(), \n#     epochs=5, \n#     callbacks=[lr_schedule, model_checkpoint_callback_efnB0],\n#     steps_per_epoch=NUM_TRAINING_IMAGES // BATCH_SIZE,\n#     validation_data=get_validation_dataset()\n# )","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# model_efn_b0.load_weights('/kaggle/working/model_efnB0_best_val_acc.hdf5')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# history_efn_b1 = model_efn_b1.fit(\n#     get_training_dataset(), \n#     epochs=5, \n#     callbacks=[lr_schedule, model_checkpoint_callback_efnB1],\n#     steps_per_epoch=NUM_TRAINING_IMAGES // BATCH_SIZE,\n#     validation_data=get_validation_dataset()\n# )","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# model_efn_b1.load_weights('/kaggle/working/model_efnB1_best_val_acc.hdf5')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# history_efn_b3 = model_efn_b3.fit(\n#     get_training_dataset(), \n#     epochs=EPOCHS, \n#     callbacks=[lr_schedule, model_checkpoint_callback_efnB3],\n#     steps_per_epoch=NUM_TRAINING_IMAGES // BATCH_SIZE,\n#     validation_data=get_validation_dataset()\n# )","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# model_efn_b3.load_weights('/kaggle/working/model_efnB3_best_val_auc.hdf5')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 11. Prediction and Submission Generation","metadata":{}},{"cell_type":"code","source":"test_ds = get_test_dataset(ordered=True)\n\nprint('Computing predictions...')\ntest_images_ds = test_ds.map(lambda image, idnum: image)\nprobabilities = model_efn_b0.predict(test_images_ds)\n\n\nprint('Generating submission.csv file...')\ntest_ids_ds = test_ds.map(lambda image, idnum: idnum).unbatch()\ntest_ids = next(iter(test_ids_ds.batch(NUM_TEST_IMAGES))).numpy().astype('U') # all in one batch\n\npred_df = pd.DataFrame({'image_name': test_ids, 'target': np.concatenate(probabilities)})\npred_df.head()\n\n\ndel sub['target']\nsub = sub.merge(pred_df, on='image_name')\nsub.to_csv('submission_label_smoothing.csv', index=False)\nsub.to_csv('submission.csv', index=False)\nsub.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}