{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Let's Discuss different types of models used\n**Here I used basic three types models and I would explain why I used it**\n\nEfficient-net is the most used model in this competition and from observation B3-B7 model is giving the best result, somehow\nadding more dense layers is reducing the LB. So I planned to add a residual block which increased my Public LB(+0.10)\n, adding dropout is a good idea. I would do some changes though like the alpha in focal loss 0.7-0.8.\n\n[Check this discussion](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/165352)\n\nSo the three models are:\n* Basic model\n* Dense added model\n* Residual block added model\n\nAny more suggestions are welcome.\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Importing Library Files","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"\n!pip install -q efficientnet\nimport numpy as np\nimport pandas as pd \nimport efficientnet.tfkeras as efn\nfrom tensorflow.keras import backend as K\nimport tensorflow_addons as tfa\nimport tensorflow.keras.layers as layers\nimport tensorflow as tf\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"img_size=384","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Binary Focal Loss \nChanged alpha from 0.25 to 0.75 since the imbalanced dataset","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def binary_focal_loss(gamma=2., alpha=.75):\n    \"\"\"\n    Binary form of focal loss.\n      FL(p_t) = -alpha * (1 - p_t)**gamma * log(p_t)\n      where p = sigmoid(x), p_t = p or 1 - p depending on if the label is 1 or 0, respectively.\n    References:\n        https://arxiv.org/pdf/1708.02002.pdf\n    Usage:\n     model.compile(loss=[binary_focal_loss(alpha=.25, gamma=2)], metrics=[\"accuracy\"], optimizer=adam)\n    \"\"\"\n    def binary_focal_loss_fixed(y_true, y_pred):\n        \"\"\"\n        :param y_true: A tensor of the same shape as `y_pred`\n        :param y_pred:  A tensor resulting from a sigmoid\n        :return: Output tensor.\n        \"\"\"\n        pt_1 = tf.where(tf.equal(y_true, 1), y_pred, tf.ones_like(y_pred))\n        pt_0 = tf.where(tf.equal(y_true, 0), y_pred, tf.zeros_like(y_pred))\n\n        epsilon = K.epsilon()\n        # clip to prevent NaN's and Inf's\n        pt_1 = K.clip(pt_1, epsilon, 1. - epsilon)\n        pt_0 = K.clip(pt_0, epsilon, 1. - epsilon)\n\n        return -K.sum(alpha * K.pow(1. - pt_1, gamma) * K.log(pt_1)) \\\n               -K.sum((1 - alpha) * K.pow(pt_0, gamma) * K.log(1. - pt_0))\n\n    return binary_focal_loss_fixed","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Basic Model\n\nEfficient Net Architecture \n\n![](https://gitcdn.xyz/cdn/Tony607/blog_statics/36894ad880dc3e645513efc36cc070c4cd0d3d7c/images/efficientnet/building_blocks.png)\n\n","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"\ndef basic_model():\n    inp=layers.Input(shape=(img_size,img_size,3),name='inp')\n    efnetb3 = efn.EfficientNetB3(weights = 'imagenet', include_top = False)\n    x=efnetb3(inp)\n    output=layers.Dense(1,activation='sigmoid')(x)\n    model = tf.keras.models.Model(inputs = [inp], outputs = [output])\n    opt = tf.keras.optimizers.Adam(learning_rate = LR)\n        # opt = tfa.optimizers.SWA(opt)\n    model.compile(optimizer = opt,loss = [binary_focal_loss(gamma = 2.0, alpha = 0.80)],metrics = [tf.keras.metrics.BinaryAccuracy(), tf.keras.metrics.AUC()])\n    \n    return model\n    \n\n    \n    \n    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Dense Added Model**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def dense_added_model():\n    inp=layers.Input(shape=(img_size,img_size,3),name='inp')\n    efnetb3 = efn.EfficientNetB3(weights = 'imagenet', include_top = False)\n    x=efnetb3(inp)\n    x=layers.Dense(256,activation='relu')(x)\n    x=layers.Dropout(0.6)(x)\n    x=layers.Dense(128,activation='relu')(x)\n    x=layers.Dropout(0.6)(x)\n    x=layers.Dense(64,activation='relu')(x)\n    x=layers.Dropout(0.3)(x)\n    output=layers.Dense(1,activation='sigmoid')(x)\n    model = tf.keras.models.Model(inputs = [inp], outputs = [output])\n    opt = tf.keras.optimizers.Adam(learning_rate = LR)\n        # opt = tfa.optimizers.SWA(opt)\n    model.compile(optimizer = opt,loss = [binary_focal_loss(gamma = 2.0, alpha = 0.80)],metrics = [tf.keras.metrics.BinaryAccuracy(), tf.keras.metrics.AUC()])\n    \n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Residual Block\n\nResidual Block Architecture , I have added leaky relu instead relu.\n\n![](https://www.researchgate.net/profile/Nazneen_Sultana/publication/326372957/figure/fig1/AS:714646050832384@1547396304214/Illustration-of-a-typical-residual-block-of-ResNet-50-layers-where-each-layer-consists.ppm)\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def residual_block(y,nb_channels_in,nb_channels_out,strides=(1,1)):\n    def conv_block(feat_maps_out, prev):\n        y = layers.BatchNormalization(prev)\n        y = layers.LeakyReLU()(y)\n        y = layers.Conv2D(nb_channels, kernel_size=(3, 3), strides=_strides, padding='same')(y)\n        y = layers.BatchNormalization()(y)\n        y = layers.LeakyReLU()(y)\n        y = layers.Conv2D(nb_channels, kernel_size=(3, 3), strides=_strides, padding='same')(y)\n        return y\n    def skip_block(feat_maps_in, feat_maps_out, prev):\n        if feat_maps_in != feat_maps_out:\n        # This adds in a 1x1 convolution on shortcuts that map between an uneven amount of channels\n        prev = layers.Conv2D(feat_maps_out,kernel_size=(1, 1), padding='same')(prev)\n        return prev \n    '''\n    A customizable residual unit with convolutional and shortcut blocks\n    Args:\n      feat_maps_in: number of channels/filters coming in, from input or previous layer\n      feat_maps_out: how many output channels/filters this block will produce\n      prev_layer: the previous layer\n    '''\n\n    skip = skip_block(nb_channels_in,nb_channels_out, y)\n    conv = conv_block(nb_channels_out,y)\n    \n    merger=layers.add([skip, conv])\n    output = layers.LeakyReLU()(merger)\n    return output\n    \ndef eff_res():\n    inp=layers.Input(shape=(img_size,img_size,3),name='inp')\n    efnetb3 = efn.EfficientNetB3(weights = 'imagenet', include_top = False)\n    x=efnetb3(inp)\n    x=layers.GlobalAveragePooling2D()(x)\n    x=residual_block(x,1536,512)\n    x= layers.AveragePooling2D(pool_size=(4, 4))(x)\n    x=layers.Flatten()(x)\n    output=layers.Dense(1,activation='sigmoid')(x)\n    model = tf.keras.models.Model(inputs = [inp], outputs = [output])\n    opt = tf.keras.optimizers.Adam(learning_rate = LR)\n        # opt = tfa.optimizers.SWA(opt)\n    model.compile(optimizer = opt,loss = [binary_focal_loss(gamma = 2.0, alpha = 0.80)],metrics = [tf.keras.metrics.BinaryAccuracy(), tf.keras.metrics.AUC()])\n    return model\n    \n    \n    \n\n \n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}