{"cells":[{"metadata":{},"cell_type":"markdown","source":"in this notebook i will use this awesome notebook [Incredible TPUs - finetune EffNetB0-B6 at once](https://www.kaggle.com/agentauers/incredible-tpus-finetune-effnetb0-b6-at-once) for my modeling.\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"![](https://media.cdn.wederm.com/wp-content/uploads/sites/3/2019/06/19205822/iStock-1060795860-300x208.jpg)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Hello Everyone! i hope you are well and  enjoying things on kaggle as usual. we know that DEEP LEARNING has huge scope in healthcare but unfortunately applying them in healthcare isn't that simple and the stake is very high. IMHO this competition is more than just a CLASSIFICATION problem and if applied very carefully, it can benefit the world in enormous ways. A common warning sign of melanoma is change. Melanoma often begins in or near an existing mole. A change in the shape, color or size of a mole can be a warning sign of melanoma. Also be aware if a mole becomes painful or begins to bleed or itch.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**There are four main types of melanoma, including a rare but dangerous kind that typically affects people of color.**\n\n![](https://images.agoramedia.com/everydayhealth/gcms/Melanoma-Types-722x406.jpg)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Imports","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install -q efficientnet","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport os, random, re, math, time\nrandom.seed(a=42)\nimport numpy as np\nimport pandas as pd\nimport tensorflow as tf\nimport tensorflow.keras.backend as K\n\n\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Conv2D, MaxPooling2D, Dense, Dropout, Input, Flatten, SeparableConv2D\nfrom tensorflow.keras.layers import GlobalMaxPooling2D\nfrom tensorflow.keras.layers import BatchNormalization\nfrom tensorflow.keras.layers import Concatenate\nfrom tensorflow.keras.models import Model\nfrom tensorflow.keras.optimizers import Adam, SGD, RMSprop\nfrom tensorflow.keras.callbacks import ModelCheckpoint, Callback, EarlyStopping\nfrom tensorflow.keras.utils import to_categorical\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import StandardScaler\nfrom mlxtend.plotting import plot_confusion_matrix\nfrom sklearn.metrics import confusion_matrix\nfrom sklearn.model_selection import KFold\nfrom sklearn.metrics import roc_auc_score\nimport cv2\nfrom tensorflow.keras import backend as K\nimport seaborn as sns\ncolor = sns.color_palette()\n%matplotlib inline\n\nfrom skimage.io import imread\nimport efficientnet.tfkeras as efn\nimport PIL\nfrom kaggle_datasets import KaggleDatasets\nfrom tqdm import tqdm\nimport matplotlib.pyplot as  plt\nimport seaborn as sns","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tf.__version__ ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"BASEPATH = \"../input/siim-isic-melanoma-classification\"\ndf_train = pd.read_csv(os.path.join(BASEPATH, 'train.csv'))\ndf_test  = pd.read_csv(os.path.join(BASEPATH, 'test.csv'))\ndf_sub   = pd.read_csv(os.path.join(BASEPATH, 'sample_submission.csv'))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# How many samples for each class are there in the dataset?","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Get the counts for each class\ncases_count = df_train['target'].value_counts()\nprint(cases_count)\n\n# Plot the results \nplt.figure(figsize=(10,8))\nsns.barplot(x=cases_count.index, y= cases_count.values)\nplt.title('Number of cases', fontsize=14)\nplt.xlabel('Case type', fontsize=12)\nplt.ylabel('Count', fontsize=12)\nplt.xticks(range(len(cases_count.index)), ['Normal(0)', 'melanoma(1)'])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"as you can see the data is highly imbalance where only 584 positive samples and 32542 negative sample images were given so it is very easy for models to mark almost any given image as non melanoma sample and if it does that then we don't get any help from our Deep Learning model to solve this healthcare related problem,right? IMHO the main goal of a healthcare related problem while using Deep Learning / Machine learning for solving that is to **REDUCE FALSE POSITIVE(FP) RATE** .A false positive is an outcome where the model incorrectly predicts the positive class.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Now Let's Know a Little About Melanoma","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"![](https://oblique.co.nz/images/hair_health/abcdes.jpg)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"[**Identifying Melanoma**](https://oblique.co.nz/hair_health/melanoma.html)\n\nThe A.B.C.D. rule has been very effective at aiding the early identification of superficial spreading melanomas. Superficial spreading melanoma can have any one of the following criteria.\n\n- Asymmetry – The shape of one half does not match the other.\n- Border – The edges are often ragged, notched, blurred, or irregular in outline; the pigment may spread into the surrounding skin.\n- Colour – The colour is uneven. Shades of black, brown, and tan may be present. Areas of white, grey, red, pink, or blue also may be seen.\n- Diameter – Size changes and usually increases. Typically, melanomas are at least 6mm in diameter (the diameter of a pencil).","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"[**Deadly Melanoma Risk Factors**](https://oblique.co.nz/hair_health/melanoma.html)\n\n*Many risk factors for melanoma have been identified. The most important risk factor is exposure to sunlight, particularly UV-B radiation. Recent studies suggest that cutaneous melanomas of the head and neck are significantly more likely to occur in people with high levels of total sun exposure. Conversely, melanomas on the trunk tend to develop on people with lower levels of ambient sunlight exposure, but who also experience higher levels of recreational exposure on the chest and back. Sunburns early in life and exposure to UV radiation from tanning beds are other factors in the development of melanoma.* **People who burn easily, such as those with fair or red hair, blue eyes, and light-coloured skin, are most prone to develop melanoma. The presence of freckling  also indicates an increased risk for melanoma development**\n\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**[People with these risk factors have a higher probability of developing melanoma:](https://www.wederm.com/2019/06/05/the-abcde-rules-of-melanoma/)**\n\n1. Fair, sun-sensitive skin that burns easily or tans poorly\n2. Red or blond hair; Blue or green eyes\n3. Have 50 or more moles\n4. Unusual or irregular-looking moles that are typically large in size\n5. A history of sunburns or indoor tanning bed use\n6. Blood relatives who have or had melanoma\n7. Immune system weakness due to disease, organ transplant or medication\n8. History of melanoma or another skin cancer\n9. Age 50 or older","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"[**Most common site for melanoma**](https://www.mdedge.com/dermatology/article/203885/melanoma/infographic-skin-cancer-stats-hispanic-patients)\n\n![](https://iuploads.scribblecdn.net/afb5a541-2cdc-445c-a47d-49d96ed3bdf4/global/imagelib/ct_skin_cancer/revisedparts.gif)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Early detection and diagnosing melanoma can save life sometimes, if you read this [Woman diagnosed with melanoma twice in her 20s warns tanning is 'not worth it'](https://www.today.com/health/how-spot-melanoma-woman-diagnosed-skin-cancer-twice-30-t180904) then you will know that a melanoma survivor Elizabeth Hazuka saying “Getting skin checks is so important,\"\n\n![](https://media4.s-nbcnews.com/i/newscms/2020_19/1566685/skin-cancer-today-main-200506_718fb2508acc5ade6d224acb597f8dc5.jpg)\n\nfrom the right side picture you can see **Doctors removed the spot and a large margin of skin to make sure all of the cancer was gone.**\njust imagine that if it was not detected and diagnosed at early stage then what could happen?","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Facts\n\n1. Melanoma is the most serious form of skin cancer\n2. It makes up 2% of skin cancers, but is responsible for 75% of skin cancer deaths\n3. Australia and New Zealand have highest melanoma rates in the world\n4. 1 in 17 Australians will be diagnosed with melanoma before the age of 85\n5. Most melanomas are caused by prolonged and repeated exposure to UV radiation in sunlight\n6. More than 90% of melanoma can be successfully treated with surgery if detected early\n\n**These facts were collated from Melanoma Patients Australia. Head over to their [website here](https://melanomapatients.org.au/about-melanoma/melanoma-facts/) for more information.**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"[**Levels of Melanoma** ](http://runningalive.com/melanoma-skin-cancer-now-bed-bound-and-no-running/)\n\nThe Clark Scale has 5 levels:\n\n1. Cells are in the out layer of the skin (epidermis)\n2. Cells are in the layer directly under the epidermis (pupillary dermis)\n3. The cells are touching the next layer known as the deep dermis\n4. Cells have spread to the reticular dermis\n5. Cells have grown in the fat layer\n\n![](https://media.giphy.com/media/lSJElktZ5BKUvYSztq/giphy.gif)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Diagnosis\nLucky for me my melanoma is at Level 1:)\n\n\n![](http://runningalive.com/wp-content/uploads/2019/10/Clark-Scale-Level-1-Melanoma.001-480x270.jpeg)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Enough domain knowledge,now let's work on modeling section for detecting this deadly disease!","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Reading melanoma data,thank you Chris Deotte :)","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"GCS_PATH    = KaggleDatasets().get_gcs_path('melanoma-256x256')\nGCS_PATH2    = KaggleDatasets().get_gcs_path('isic2019-256x256')\nfiles_train = tf.io.gfile.glob(GCS_PATH + '/train*.tfrec')\n#files_train += tf.io.gfile.glob(GCS_PATH2 + '/train*.tfrec')\nfiles_train1 = tf.io.gfile.glob(GCS_PATH2 + '/train*.tfrec')\nfiles_test  = np.sort(np.array(tf.io.gfile.glob(GCS_PATH + '/test*.tfrec')))\n#np.random.shuffle(files_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model configuration","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"DEVICE = \"TPU\"\nCFG = dict(\n    batch_size        =  128,\n    read_size         = 256, \n    crop_size         = 250, \n    net_size          = 248,\n    LR_START          =   0.00005,\n    LR_MAX            =   0.00020,\n    LR_MIN            =   0.00001,\n    LR_RAMPUP_EPOCHS  =   5,\n    LR_SUSTAIN_EPOCHS =   0,\n    LR_EXP_DECAY      =   0.8,\n    \n    epochs            =  5,\n    \n    rot               = 180.0,\n    shr               =   1.5,\n    hzoom             =   6.0,\n    wzoom             =   6.0,\n    hshift            =   6.0,\n    wshift            =   6.0,\n    optimizer         = 'adam',\n    label_smooth_fac  =   0.05,\n    tta_steps         =  25    \n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Tpu setting","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"if DEVICE == \"TPU\":\n    print(\"connecting to TPU...\")\n    try:\n        tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n        print('Running on TPU ', tpu.master())\n    except ValueError:\n        print(\"Could not connect to TPU\")\n        tpu = None\n\n    if tpu:\n        try:\n            print(\"initializing  TPU ...\")\n            tf.config.experimental_connect_to_cluster(tpu)\n            tf.tpu.experimental.initialize_tpu_system(tpu)\n            strategy = tf.distribute.experimental.TPUStrategy(tpu)\n            print(\"TPU initialized\")\n        except _:\n            print(\"failed to initialize TPU\")\n    else:\n        DEVICE = \"GPU\"\n\nif DEVICE != \"TPU\":\n    print(\"Using default strategy for CPU and single GPU\")\n    strategy = tf.distribute.get_strategy()\n\nif DEVICE == \"GPU\":\n    print(\"Num GPUs Available: \", len(tf.config.experimental.list_physical_devices('GPU')))\n    \n\nAUTO     = tf.data.experimental.AUTOTUNE\nREPLICAS = strategy.num_replicas_in_sync\nprint(f'REPLICAS: {REPLICAS}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_mat(rotation, shear, height_zoom, width_zoom, height_shift, width_shift):\n    # returns 3x3 transformmatrix which transforms indicies\n        \n    # CONVERT DEGREES TO RADIANS\n    rotation = math.pi * rotation / 180.\n    shear    = math.pi * shear    / 180.\n\n    def get_3x3_mat(lst):\n        return tf.reshape(tf.concat([lst],axis=0), [3,3])\n    \n    # ROTATION MATRIX\n    c1   = tf.math.cos(rotation)\n    s1   = tf.math.sin(rotation)\n    one  = tf.constant([1],dtype='float32')\n    zero = tf.constant([0],dtype='float32')\n    \n    rotation_matrix = get_3x3_mat([c1,   s1,   zero, \n                                   -s1,  c1,   zero, \n                                   zero, zero, one])    \n    # SHEAR MATRIX\n    c2 = tf.math.cos(shear)\n    s2 = tf.math.sin(shear)    \n    \n    shear_matrix = get_3x3_mat([one,  s2,   zero, \n                                zero, c2,   zero, \n                                zero, zero, one])        \n    # ZOOM MATRIX\n    zoom_matrix = get_3x3_mat([one/height_zoom, zero,           zero, \n                               zero,            one/width_zoom, zero, \n                               zero,            zero,           one])    \n    # SHIFT MATRIX\n    shift_matrix = get_3x3_mat([one,  zero, height_shift, \n                                zero, one,  width_shift, \n                                zero, zero, one])\n    \n    return K.dot(K.dot(rotation_matrix, shear_matrix), \n                 K.dot(zoom_matrix,     shift_matrix))\n\n\ndef transform(image, cfg):    \n    # input image - is one image of size [dim,dim,3] not a batch of [b,dim,dim,3]\n    # output - image randomly rotated, sheared, zoomed, and shifted\n    DIM = cfg[\"read_size\"]\n    XDIM = DIM%2 #fix for size 331\n    \n    rot = cfg['rot'] * tf.random.normal([1], dtype='float32')\n    shr = cfg['shr'] * tf.random.normal([1], dtype='float32') \n    h_zoom = 1.0 + tf.random.normal([1], dtype='float32') / cfg['hzoom']\n    w_zoom = 1.0 + tf.random.normal([1], dtype='float32') / cfg['wzoom']\n    h_shift = cfg['hshift'] * tf.random.normal([1], dtype='float32') \n    w_shift = cfg['wshift'] * tf.random.normal([1], dtype='float32') \n\n    # GET TRANSFORMATION MATRIX\n    m = get_mat(rot,shr,h_zoom,w_zoom,h_shift,w_shift) \n\n    # LIST DESTINATION PIXEL INDICES\n    x   = tf.repeat(tf.range(DIM//2, -DIM//2,-1), DIM)\n    y   = tf.tile(tf.range(-DIM//2, DIM//2), [DIM])\n    z   = tf.ones([DIM*DIM], dtype='int32')\n    idx = tf.stack( [x,y,z] )\n    \n    # ROTATE DESTINATION PIXELS ONTO ORIGIN PIXELS\n    idx2 = K.dot(m, tf.cast(idx, dtype='float32'))\n    idx2 = K.cast(idx2, dtype='int32')\n    idx2 = K.clip(idx2, -DIM//2+XDIM+1, DIM//2)\n    \n    # FIND ORIGIN PIXEL VALUES           \n    idx3 = tf.stack([DIM//2-idx2[0,], DIM//2-1+idx2[1,]])\n    d    = tf.gather_nd(image, tf.transpose(idx3))\n        \n    return tf.reshape(d,[DIM, DIM,3])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def read_labeled_tfrecord(example):\n    tfrec_format = {\n        'image'                        : tf.io.FixedLenFeature([], tf.string),\n        'image_name'                   : tf.io.FixedLenFeature([], tf.string),\n        'patient_id'                   : tf.io.FixedLenFeature([], tf.int64),\n        'sex'                          : tf.io.FixedLenFeature([], tf.int64),\n        'age_approx'                   : tf.io.FixedLenFeature([], tf.int64),\n        'anatom_site_general_challenge': tf.io.FixedLenFeature([], tf.int64),\n        'diagnosis'                    : tf.io.FixedLenFeature([], tf.int64),\n        'target'                       : tf.io.FixedLenFeature([], tf.int64)\n    }           \n    example = tf.io.parse_single_example(example, tfrec_format)\n    return example['image'], example['target']\n\n\ndef read_unlabeled_tfrecord(example, return_image_name):\n    tfrec_format = {\n        'image'                        : tf.io.FixedLenFeature([], tf.string),\n        'image_name'                   : tf.io.FixedLenFeature([], tf.string),\n    }\n    example = tf.io.parse_single_example(example, tfrec_format)\n    return example['image'], example['image_name'] if return_image_name else 0\n\n \ndef prepare_image(img, cfg=None, augment=True):    \n    img = tf.image.decode_jpeg(img, channels=3)\n    img = tf.image.resize(img, [cfg['read_size'], cfg['read_size']])\n    img = tf.cast(img, tf.float32) / 255.0\n    \n    if augment:\n        img = transform(img, cfg)\n        img = tf.image.random_crop(img, [cfg['crop_size'], cfg['crop_size'], 3])\n        img = tf.image.random_flip_left_right(img)\n        img = tf.image.random_hue(img, 0.01)\n        img = tf.image.random_saturation(img, 0.7, 1.3)\n        img = tf.image.random_contrast(img, 0.8, 1.2)\n        img = tf.image.random_brightness(img, 0.1)\n\n    else:\n        img = tf.image.central_crop(img, cfg['crop_size'] / cfg['read_size'])\n                                   \n    img = tf.image.resize(img, [cfg['net_size'], cfg['net_size']])\n    img = tf.reshape(img, [cfg['net_size'], cfg['net_size'], 3])\n    return img\n\ndef count_data_items(filenames):\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) \n         for filename in filenames]\n    return np.sum(n)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_dataset(files, cfg, augment = False, shuffle = False, repeat = False, \n                labeled=True, return_image_names=True):\n    \n    ds = tf.data.TFRecordDataset(files, num_parallel_reads=AUTO)\n    ds = ds.cache()\n    \n    if repeat:\n        ds = ds.repeat()\n    \n    if shuffle: \n        ds = ds.shuffle(1024*8)\n        opt = tf.data.Options()\n        opt.experimental_deterministic = False\n        ds = ds.with_options(opt)\n        \n    if labeled: \n        ds = ds.map(read_labeled_tfrecord, num_parallel_calls=AUTO)\n    else:\n        ds = ds.map(lambda example: read_unlabeled_tfrecord(example, return_image_names), \n                    num_parallel_calls=AUTO)      \n    \n    ds = ds.map(lambda img, imgname_or_label: (prepare_image(img, augment=augment, cfg=cfg), \n                                               imgname_or_label), \n                num_parallel_calls=AUTO)\n    \n    ds = ds.batch(cfg['batch_size'] * REPLICAS)\n    ds = ds.prefetch(AUTO)\n    return ds","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Let's look at how a normal case is different from that of a melanoma case. We will look at somes samples from external data of alex shonenkov.**\n\nthank you alex shonenkov\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Get few samples for both the classes\nshonenkovData = pd.read_csv('../input/melanoma-merged-external-data-512x512-jpeg/folds.csv')\nmelanoma_samples = (shonenkovData[shonenkovData['target']==1]['image_id'].iloc[:5]).tolist()\nnormal_samples = (shonenkovData[shonenkovData['target']==0]['image_id'].iloc[:5]).tolist()\n# Concat the data in a single list and del the above two list\nsamples = melanoma_samples + normal_samples\ndel melanoma_samples, normal_samples\nsource = \"../input/melanoma-merged-external-data-512x512-jpeg/512x512-dataset-melanoma/512x512-dataset-melanoma/\"\n# Plot the data \nf, ax = plt.subplots(2,5, figsize=(30,10))\nfor i in range(10):\n    img = imread(source + samples[i]+\".jpg\")\n    ax[i//5, i%5].imshow(img, cmap='gray')\n    if i<5:\n        ax[i//5, i%5].set_title(\"melanoma\")\n    else:\n        ax[i//5, i%5].set_title(\"Normal\")\n    ax[i//5, i%5].axis('off')\n    ax[i//5, i%5].set_aspect('auto')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"i am not a radiologist,but correct me if i am wrong, using my naked eyes it looks like some normal cases more  like melanoma cases  hence making it more difficult for models to generalize?","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**now let's have a look at competitions resized data of chris data and try to compare them with external data**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Train samples","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def show_dataset(thumb_size, cols, rows, ds):\n    mosaic = PIL.Image.new(mode='RGB', size=(thumb_size*cols + (cols-1), \n                                             thumb_size*rows + (rows-1)))\n   \n    for idx, data in enumerate(iter(ds)):\n        img, target_or_imgid = data\n        ix  = idx % cols\n        iy  = idx // cols\n        img = np.clip(img.numpy() * 255, 0, 255).astype(np.uint8)\n        img = PIL.Image.fromarray(img)\n        img = img.resize((thumb_size, thumb_size), resample=PIL.Image.BILINEAR)\n        mosaic.paste(img, (ix*thumb_size + ix, \n                           iy*thumb_size + iy))\n\n    display(mosaic)\n    \nds = get_dataset(files_train, CFG).unbatch().take(12*2)   \nshow_dataset(512, 12, 2, ds)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"ds = tf.data.TFRecordDataset(files_train, num_parallel_reads=AUTO)\nds = ds.take(1).cache().repeat()\nds = ds.map(read_labeled_tfrecord, num_parallel_calls=AUTO)\nds = ds.map(lambda img, target: (prepare_image(img, cfg=CFG, augment=True), target), \n            num_parallel_calls=AUTO)\nds = ds.take(12*2)\nds = ds.prefetch(AUTO)\n\nshow_dataset(512, 12, 2, ds)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# test samples","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"ds = get_dataset(files_test, CFG, labeled=False).unbatch().take(12*2)   \nshow_dataset(512, 12, 2, ds)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**to me it seems like external data is little bit different than our competitions data,what  do you think?**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"now let's visualize some of external data that chris deotte prepared,thanks to [this post](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/164910) i will visualize some samples from [this data](https://www.kaggle.com/cdeotte/isic2019-256x256)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\n\n# Get few samples for both the classes\nchrisData = pd.read_csv('../input/jpeg-isic2019-256x256/train.csv')\nmelanoma_samples = (chrisData[chrisData['target']==1]['image_name'].iloc[:5]).tolist()\nnormal_samples = (chrisData[chrisData['target']==0]['image_name'].iloc[:5]).tolist()\n# Concat the data in a single list and del the above two list\nsamples = melanoma_samples + normal_samples\ndel melanoma_samples, normal_samples\nsource = \"../input/jpeg-isic2019-256x256/train/\"\n# Plot the data \nf, ax = plt.subplots(2,5, figsize=(30,10))\nfor i in range(10):\n    img = imread(source + samples[i]+\".jpg\")\n    ax[i//5, i%5].imshow(img, cmap='gray')\n    if i<5:\n        ax[i//5, i%5].set_title(\"melanoma\")\n    else:\n        ax[i//5, i%5].set_title(\"Normal\")\n    ax[i//5, i%5].axis('off')\n    ax[i//5, i%5].set_aspect('auto')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Again with naked eyes i can see some normal cases looks like melanoma! your thought?","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Depthwise Separable Convolution in Xception\n\nthis is the most important section, let's start with a simple yet powerful network called Xception  \n\n\n# Understanding Depthwise Separable Convolution\n\n![](https://miro.medium.com/max/1050/1*VvBTMkVRus6bWOqrK1SlLQ.png)\n\n**Above Figure : Original Depthwise Separable Convolution**\n\nThe original depthwise separable convolution is the depthwise convolution followed by a pointwise convolution.\n\n1. Depthwise convolution is the channel-wise n×n spatial convolution. Suppose in the figure above, we have 5 channels, then we will have 5 n×n spatial convolution.\n\n2. Pointwise convolution actually is the 1×1 convolution to change the dimension.\n\nCompared with conventional convolution, we do not need to perform convolution across all channels. That means the number of connections are fewer and the model is lighter.\n\n\n![](https://miro.medium.com/max/1050/1*J8dborzVBRBupJfvR7YhuA.png)\n\n**The Modified Depthwise Separable Convolution used as an Inception Module in Xception, so called “extreme” version of Inception module (n=3 here)**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"The modified depthwise separable convolution is the pointwise convolution followed by a depthwise convolution. This modification is motivated by the inception module in Inception-v3 that 1×1 convolution is done first before any n×n spatial convolutions. Thus, it is a bit different from the original one. (n=3 here since 3×3 spatial convolutions are used in Inception-v3.)\n\n**Two minor differences:**\n1. The order of operations: As mentioned, the original depthwise separable convolutions as usually implemented (e.g. in TensorFlow) perform first channel-wise spatial convolution and then perform 1×1 convolution whereas the modified depthwise separable convolution perform 1×1 convolution first then channel-wise spatial convolution. This is claimed to be unimportant because when it is used in stacked setting, there are only small differences appeared at the beginning and at the end of all the chained inception modules.\n\n2. The Presence/Absence of Non-Linearity: In the original Inception Module, there is non-linearity after first operation. In Xception, the modified depthwise separable convolution, there is NO intermediate ReLU non-linearity.\n\n![](https://miro.medium.com/max/773/1*x99M0RaK9i2X9kYrEYtYdg.png)\n\nThe modified depthwise separable convolution with different activation units are tested. As from the above figure, the Xception without any intermediate activation has the highest accuracy compared with the ones using either ELU or ReLU.\n\ninformation source : [Review: Xception — With Depthwise Separable Convolution, Better Than Inception-v3 (Image Classification)](https://towardsdatascience.com/review-xception-with-depthwise-separable-convolution-better-than-inception-v3-image-dc967dd42568)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# callback - Learning rate finder","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_lr_callback(cfg):\n    lr_start   = cfg['LR_START']\n    lr_max     = cfg['LR_MAX'] * strategy.num_replicas_in_sync\n    lr_min     = cfg['LR_MIN']\n    lr_ramp_ep = cfg['LR_RAMPUP_EPOCHS']\n    lr_sus_ep  = cfg['LR_SUSTAIN_EPOCHS']\n    lr_decay   = cfg['LR_EXP_DECAY']\n   \n    def lrfn(epoch):\n        if epoch < lr_ramp_ep:\n            lr = (lr_max - lr_start) / lr_ramp_ep * epoch + lr_start\n            \n        elif epoch < lr_ramp_ep + lr_sus_ep:\n            lr = lr_max\n            \n        else:\n            lr = (lr_max - lr_min) * lr_decay**(epoch - lr_ramp_ep - lr_sus_ep) + lr_min\n            \n        return lr\n\n    lr_callback = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose=False)\n    return lr_callback","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"[**Multi-Sample Dropout**](https://www.kaggle.com/doanquanvietnamca/multi-dropout-kfold-with-roberta-on-gpu)\n\nDropout is an efficient regularization instrument for avoiding overfitting of deep neural networks. It works very simply randomly discarding a portion of neurons during training; as a result, a generalization occurs because in this way neurons depend no more on each other. In this post, I try to reproduce the results presented in this paper; which introduced a technique called Multi-Sample Dropout. As declared by the author, its scopes are:\n\n* accelerate training and improve generalization over the original dropout\n\n* reduce computational cost because most of the computation time is consumed in the layers below (often convolutional or recurrent) and the weights in the layers at the top are shared\n\n* achieve lower error rates and losses.\n\n![](https://miro.medium.com/max/700/1*mdBXp3-D7G7mTcKDZHui8g.png)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"When a particular problem includes an imbalanced dataset, then accuracy isn't a good metric to look for. For example, if your dataset contains 95 negatives and 5 positives, having a model with 95% accuracy doesn't make sense at all. The classifier might label every example as negative and still achieve 95% accuracy. Hence, we need to look for alternative metrics. Precision and Recall are really good metrics for such kind of problems.\n\nWe will get the confusion matrix from our predictions and see what is the recall and precision of our model.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Model : will use VGG-16 from [Beating everything with Depthwise Convolution](https://www.kaggle.com/aakashnain/beating-everything-with-depthwise-convolution)\n\nthank you @aakashnain :)\n\ni really loved this [**Skin Lesion Detection**](https://github.com/AakashKumarNain/skin_cancer_detection) android app of yours :) \n\n\nThis is the best part. If you look at other kernels on this dataset, everyone is busy doing transfer learning and fine-tuning. You should transfer learn but wisely. We will be doing partial transfer learning and rest of the model will be trained from scratch. I will explain this in detail but before that, I would love to share one of the best practices when it comes to building deep learning models from scratch on limited data.\n\n1. Choose a simple architecture.\n\n2. Initialize the first few layers from a network that is pretrained on imagenet. This is because first few layers capture general details like color blobs, patches, edges, etc. Instead of randomly initialized weights for these layers, it would be much better if you fine tune them.\n\n3. Choose layers that introduce a lesser number of parameters. For example, Depthwise SeparableConv is a good replacement for Conv layer. It introduces lesser number of parameters as compared to normal convolution and as different filters are applied to each channel, it captures more information. Xception a powerful network, is built on top of such layers only. You can read about Xception and Depthwise Separable Convolutions in [this paper.](https://arxiv.org/abs/1610.02357)\n\n4. Use batch norm with convolutions. As the network becomes deeper, batch norm start to play an important role.\n\n5. Add multi sample dropout. Train with a higher learning rate and experiment with the number of neurons in the dense layers. Do it for the depth of your network too.\n\n6. Once you know a good depth, start training your network with a lower learning rate along with decay.\n\nThis is all that I have done in the next code block.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**update : i changed maxpooling layer and dropouts with multisample dropouts**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#https://www.kaggle.com/aakashnain/beating-everything-with-depthwise-convolution\ndef build_model():\n    input_img = Input(shape=(256,256,3), name='ImageInput')\n    x = Conv2D(64, (3,3), activation='relu', padding='same', name='Conv1_1')(input_img)\n    x = Conv2D(64, (3,3), activation='relu', padding='same', name='Conv1_2')(x)\n    x = MaxPooling2D((2,2), name='pool1')(x)\n    \n    x = SeparableConv2D(128, (3,3), activation='relu', padding='same', name='Conv2_1')(x)\n    x = SeparableConv2D(128, (3,3), activation='relu', padding='same', name='Conv2_2')(x)\n    x = MaxPooling2D((2,2), name='pool2')(x)\n    \n    x = SeparableConv2D(256, (3,3), activation='relu', padding='same', name='Conv3_1')(x)\n    x = BatchNormalization(name='bn1')(x)\n    x = SeparableConv2D(256, (3,3), activation='relu', padding='same', name='Conv3_2')(x)\n    x = BatchNormalization(name='bn2')(x)\n    x = SeparableConv2D(256, (3,3), activation='relu', padding='same', name='Conv3_3')(x)\n    x = MaxPooling2D((2,2), name='pool3')(x)\n    \n    x = SeparableConv2D(512, (3,3), activation='relu', padding='same', name='Conv4_1')(x)\n    x = BatchNormalization(name='bn3')(x)\n    x = SeparableConv2D(512, (3,3), activation='relu', padding='same', name='Conv4_2')(x)\n    x = BatchNormalization(name='bn4')(x)\n    x = SeparableConv2D(512, (3,3), activation='relu', padding='same', name='Conv4_3')(x)\n    x = MaxPooling2D((3,3), name='pool4')(x)\n    \n    x = Flatten(name='flatten')(x)\n\n\n    dense = []\n    for p in np.linspace(0.5,0.7, 5):\n        x = Dense(1024, activation='relu', name= f'fc1{p}')(x)\n        x_ = tf.keras.layers.Dropout(p)(x)\n        dense.append(x_)\n    x = tf.keras.layers.Average()(dense)\n    \n\n    dense = []\n    for p in np.linspace(0.3,0.5, 5):\n        x = Dense(512, activation='relu', name= f'fc2{p}')(x)\n        x_ = tf.keras.layers.Dropout(p)(x)\n        dense.append(x_)\n    x = tf.keras.layers.Average()(dense)\n    \n    \n    x = Dense(1, activation='sigmoid', name='fc3')(x)\n    \n    model = Model(inputs=input_img, outputs=x)\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model =  build_model()\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We will initialize the weights of first two convolutions with imagenet weights,","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Open the VGG16 weight file\n#https://www.kaggle.com/aakashnain/beating-everything-with-depthwise-convolution\nimport h5py\nf = h5py.File('../input/vgg16/vgg16_weights_tf_dim_ordering_tf_kernels_notop.h5', 'r')\n\n# Select the layers for which you want to set weight.\n\nw,b = f['block1_conv1']['block1_conv1_W_1:0'], f['block1_conv1']['block1_conv1_b_1:0']\nmodel.layers[1].set_weights = [w,b]\n\nw,b = f['block1_conv2']['block1_conv2_W_1:0'], f['block1_conv2']['block1_conv2_b_1:0']\nmodel.layers[2].set_weights = [w,b]\n\nw,b = f['block2_conv1']['block2_conv1_W_1:0'], f['block2_conv1']['block2_conv1_b_1:0']\nmodel.layers[4].set_weights = [w,b]\n\nw,b = f['block2_conv2']['block2_conv2_W_1:0'], f['block2_conv2']['block2_conv2_b_1:0']\nmodel.layers[5].set_weights = [w,b]\n\nf.close()\nmodel.summary()  ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.compile(\n            optimizer = CFG['optimizer'],\n            loss      =  tf.keras.losses.BinaryCrossentropy(label_smoothing = CFG['label_smooth_fac']),\n            metrics   = [tf.keras.metrics.AUC(name='auc')])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Running for 40 epochs/fold","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n# from : https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords/comments?\nskf = KFold(n_splits=2,shuffle=True,random_state=42)\noof_pred = []; oof_tar = []; oof_val = []; oof_names = [] \n\nGCS_PATH = [None]*2; IMG_SIZES = [256,256]\n\nfor i,k in enumerate(IMG_SIZES):\n    GCS_PATH[i] = KaggleDatasets().get_gcs_path('melanoma-%ix%i'%(k,k))\n    #GCS_PATH2[i] = KaggleDatasets().get_gcs_path('isic2019-%ix%i'%(k,k))\nfiles_train = np.sort(np.array(tf.io.gfile.glob(GCS_PATH[0] + '/train*.tfrec')))\nfiles_test  = np.sort(np.array(tf.io.gfile.glob(GCS_PATH[0] + '/test*.tfrec')))\n\nfor fold,(idxT,idxV) in enumerate(skf.split(np.arange(15))):\n    \n    # DISPLAY FOLD INFO\n    if DEVICE=='TPU':\n        if tpu: tf.tpu.experimental.initialize_tpu_system(tpu)\n    print('_'*25); print('**** FOLD',fold+1)\n    # CREATE TRAIN AND VALIDATION SUBSETS\n    files_train = tf.io.gfile.glob([GCS_PATH[fold] + '/train%.2i*.tfrec'%x for x in idxT])\n    files_valid = tf.io.gfile.glob([GCS_PATH[fold] + '/train%.2i*.tfrec'%x for x in idxV])\n    \n    # SAVE BEST MODEL EACH FOLD\n    sv = tf.keras.callbacks.ModelCheckpoint(\n        'vgg16fold-%i.h5'%fold, monitor='val_loss', verbose=1, save_best_only=True,\n        save_weights_only=True, mode='min', save_freq='epoch')\n    \n    steps_per_epoch  = count_data_items(files_train) / (CFG['batch_size'] * REPLICAS)\n\n    history_Xception1   = model.fit(\n        get_dataset(files_train, CFG,augment=True, shuffle=True, repeat=True), \n        epochs=40, callbacks = [sv,get_lr_callback(CFG)], \n        steps_per_epoch=count_data_items(files_train)/128//REPLICAS,\n        validation_data=get_dataset(files_valid,CFG,augment=False,shuffle=False,\n                repeat=False), \n        verbose=1\n    )\n    # PREDICT OOF USING TTA\n    TTA = 2\n    print('Predicting OOF with TTA...')\n    ds_valid = get_dataset(files_valid,CFG,labeled=False,return_image_names=False,augment=True,\n            repeat=True,shuffle=False,)\n    ct_valid = count_data_items(files_valid); \n    STEPS = TTA * ct_valid/128/REPLICAS\n    pred = model.predict(ds_valid,steps=STEPS,verbose=1)[:TTA*ct_valid,] \n\n    oof_pred.append( np.mean(pred.reshape((ct_valid,TTA),order='F'),axis=1) )                 \n\n    # GET OOF TARGETS AND NAMES\n    ds_valid = get_dataset(files_valid,CFG, augment=False, repeat=False,\n            labeled=True, return_image_names=True)\n    oof_tar.append( np.array([target.numpy() for img, target in iter(ds_valid.unbatch())]) )\n\n    # REPORT RESULTS\n    auc = roc_auc_score(oof_tar[-1],oof_pred[-1])\n    oof_val.append(np.max( history_Xception1.history['val_auc'] ))\n    print('#### FOLD %i OOF AUC with TTA = %.3f, without TTA = %.3f'%(fold+1,auc,oof_val[-1]))\n    print()   \n    \n    \n    for i in range(len(oof_pred[0])):\n        if(oof_pred[0][i] > 0.5):\n            oof_pred[0][i] = 1\n        else:\n            oof_pred[0][i] = 0\n        \n    oof_preds = []\n    oof_tars = []\n    for i in range(len(oof_pred[0])):\n        oof_preds.append(oof_pred[0][i])\n        oof_tars.append(oof_tar[0][i])\n\n\n    # Get the confusion matrix\n    cm  = confusion_matrix(oof_tars, oof_preds)\n    plt.figure()\n    plot_confusion_matrix(cm,figsize=(12,8), hide_ticks=True)\n    plt.yticks(range(2), ['Normal', 'melanoma'], fontsize=16)\n    plt.show()\n    \n    # Calculate Precision and Recall\n    tn, fp, fn, tp = cm.ravel()\n\n    precision = tp/(tp+fp)\n    recall = tp/(tp+fn)\n\n    print(\"Recall of the model is {:.2f}\".format(recall))\n    print(\"Precision of the model is {:.2f}\".format(precision))\n\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# COMPUTE OVERALL OOF AUC\noof = np.concatenate(oof_pred)\ntrue = np.concatenate(oof_tar)\nauc = roc_auc_score(true,oof)\nprint('Overall OOF AUC with TTA = %.3f'%auc)\n\n# SAVE OOF TO DISK\ndf_oof = pd.DataFrame(dict( pred = oof, target=true))\ndf_oof.to_csv('oof.csv',index=False)\ndf_oof.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"oof_tars = []\n\noof_preds = []\n\nfor i in range(len(df_oof)):\n    if(df_oof.pred[i] > 0.5):\n        oof_preds.append(1)\n        oof_tars.append(df_oof.target[i])\n    else:\n        oof_preds.append(0)\n        oof_tars.append(df_oof.target[i])\n\n\n# Get the confusion matrix\ncm  = confusion_matrix(oof_tars, oof_preds)\nplt.figure()\nplot_confusion_matrix(cm,figsize=(12,8), hide_ticks=True)\nplt.yticks(range(2), ['Normal', 'melanoma'], fontsize=16)\nplt.show()\n\n# Calculate Precision and Recall\ntn, fp, fn, tp = cm.ravel()\n\nprecision = tp/(tp+fp)\nrecall = tp/(tp+fn)\n\nprint(\"Overall Recall of the model is {:.2f}\".format(recall))\nprint(\"Overall Precision of the model is {:.2f}\".format(precision))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history_Xception1.history['lr']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Predicting","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nCFG['batch_size'] = 256\n\ncnt_test   = count_data_items(files_test)\nsteps      = cnt_test / (CFG['batch_size'] * REPLICAS) * CFG['tta_steps']\nds_testAug = get_dataset(files_test, CFG, augment=True, repeat=True, \n                         labeled=False, return_image_names=False)\npreds = model.predict(ds_testAug, verbose=1, steps=steps)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Test shape :\",df_sub.shape)\nprint(\"Preds shape :\",preds.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# https://www.kaggle.com/vbhargav875/efficientnet-b5-b6-b7-tf-keras\n\npreds = preds[:,:cnt_test* CFG['tta_steps']]\npreds = preds[:df_test.shape[0]*CFG['tta_steps']]\npreds = np.stack(np.split(preds, CFG['tta_steps']),axis=1)\npreds = np.mean(preds, axis=1)\npreds = preds.reshape(-1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"New Preds shape :\",preds.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Submission","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"ds = get_dataset(files_test, CFG, augment=False, repeat=False, \n                 labeled=False, return_image_names=True)\n\nimage_names = np.array([img_name.numpy().decode(\"utf-8\") \n                        for img, img_name in iter(ds.unbatch())])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = pd.DataFrame(dict(\n        image_name = image_names,\n        target = preds ))\n\nsubmission = submission.sort_values('image_name') \nsubmission.to_csv(f'submission_vgg16.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"on tpu i get [this error](https://github.com/tensorflow/tensorflow/issues/8471)\n\nso running on gpu just to share with you all,i think training for more epochs will be good!","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}