{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d2fd95afc524740e3d1d750e6645706ba089bd75"},"cell_type":"markdown","source":"## Intro\nI'm a beginner in deeplearning, I just learned these knowledge for 3 months, and this is my first competition on kaggle. I've learned a lot from this competition, I'm really grateful for all the kernels and contributors.\n\nAs far as I can tell in this topic, there are 3 major issues that should be concerned, and these issues also differ this compettition from general image recognition problems. First, only about 30,000 data set are provided as training set, which is not enough for training a new model from scratch. So, use pre-train model would help a lot(I did tried a version that started with no pre-trained weight, which got low F1-score) . Second, these are not typical RGB images, each of them is seperated into RGBY images. Also, they are not only color itself, each color infers different structures in human cells. Lastly, this is a multi-class + multi-label mission, and there exist strong imbalance between all 28 types of protein(labels). Some labels are extremely rare while some are much mor frequently labeled.\n\nHere is my kernel for dealing with all these issues, as a record for my first DL problem."},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"import os\nimport numpy as np\nimport pandas as pd\nfrom PIL import Image\nfrom zipfile import ZipFile\nimport matplotlib.pyplot as plt\nimport cv2\nimport math\nfrom imgaug import augmenters as iaa\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ee04dc60d6f582f796b568461aa58c8aca9dd351"},"cell_type":"code","source":"os.listdir('../input')\nPATH_BASE = '../input/'\nTRAIN_BASE = 'human-protein-atlas-image-classification/'\nMODEL_BASE = '4-channel-v2-with-rare-case-extension-t2/'\nPATH_TRAIN = PATH_BASE+TRAIN_BASE+'train/'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7ac7e4c4a6e8f87c37c5bb9f27c537fa25979b54"},"cell_type":"code","source":"raw_labels = pd.read_csv(PATH_BASE+TRAIN_BASE+'train.csv')\ndata_names = os.listdir(PATH_TRAIN)\n\n#extract label names and labels array[{name: ,label:}]\nlabels = []\nfor name, label in zip(raw_labels['Id'],raw_labels['Target'].str.split(\" \")):\n    labels.append({\n        'name':name,\n        'label':label\n    })\n#Split data to train/dev set\nfrom sklearn.model_selection import train_test_split\ntrain_idx, test_idx = train_test_split(labels, test_size=0.2)\nprint('train: ' + str(len(train_idx)) + '\\n'+ 'validation: ' + str(len(test_idx)))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f31228d03eb295ce84a3671f39904e1a22f849cf"},"cell_type":"markdown","source":" # Data Visualization\n credit:\n https://www.kaggle.com/wordroid/resnet50-4x256-globalmax-lb-0-443\n"},{"metadata":{"_uuid":"6b9664c26d0df64da8f06c61dc971d51ec2c9d51"},"cell_type":"markdown","source":"In the cell below, we can observe data imbalance. \nWhile label 0 and 25 occupy 67% of the training set, label 8,9,10,15,27 only take approximately 0.001%."},{"metadata":{"trusted":true,"_uuid":"83677e2729802a26af60ada060c0a194b80b6269"},"cell_type":"code","source":"y_cat_train_dic = {}\nfor icat in range(28):\n    target = str(icat)\n    y_cat_train_5 = np.array([int(target in ee['label']) for ee in train_idx])\n    y_cat_train_dic[icat] = y_cat_train_5\nup_sample = {}\nfor k in y_cat_train_dic:\n    v = y_cat_train_dic[k].sum()\n    up_sample[k] = np.round(v / len(train_idx), 5)\nprint(up_sample)\ndef plt_barh(x, y, title):\n    fig, ax = plt.subplots(figsize=(15,7))\n    width = 0.75\n    ind = np.arange(len(up_sample))  # the x locations for the groups\n    ax.barh(ind, y, width, color=\"blue\")\n    ax.set_yticks(ind+width/2)\n    ax.set_yticklabels(x, minor=False)\n    plt.title(title)\n    for i, v in enumerate(y):\n        ax.text(v, i , str(v), color='blue', fontweight='bold')\n    plt.xlabel('x')\n    plt.ylabel('y')\nx = list(up_sample.keys())\ny = list(up_sample.values())\nplt_barh(x, y, 'data imbalance')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"91a222e94f7ddc697d2ccc9a325a91911378d3da"},"cell_type":"markdown","source":"In the cell below, four channels of images are showd. \nAlso, I tried to merge Y channel into RGB in different ways for my training set.\n\nNote: Afterwards, I chose to use 4-channels for my model. Actually, merging Y channel to train a 3-channels model also works, got about 0.4 LB score. However, since useing 3 or 4 channels doesn't show significant difference in training performance(in my case), I chose 4-channel because it's more reasonable to use as much as the informations we can access in datas."},{"metadata":{"trusted":true,"_uuid":"514f705937efc603aac2a3298151587dda758c7f"},"cell_type":"code","source":"#np.random.seed(18)\ntest = labels[10]\nprint(test)\nprint(test['name'])\nprint(test['label'])\n\nfig, ax = plt.subplots(1,4,figsize=(12,12))\nfig.tight_layout()\n#Try different mix method\nnames = [n['name'] for n in np.random.choice(labels, 1)]\nR = np.array(Image.open(PATH_TRAIN+names[0]+'_red.png'))\nax[0].imshow(R,cmap='Reds')\nax[0].set_title('R')\nG = np.array(Image.open(PATH_TRAIN+names[0]+'_green.png'))\nax[1].imshow(G,cmap='Greens')\nax[1].set_title('G')\nB = np.array(Image.open(PATH_TRAIN+names[0]+'_blue.png'))\nax[2].imshow(B,cmap='Blues')\nax[2].set_title('B')\nY = np.array(Image.open(PATH_TRAIN+names[0]+'_yellow.png'))\nax[3].imshow(Y,cmap='YlOrBr')\nax[3].set_title('Y')\n\nBY = (B+Y)\nBY[BY>255] = 255\nRY = (R+Y)\nRY[RY>255] = 255\nGY = (G+Y)\nGY[GY>255] = 255\n\nIMG = np.stack((R, G, B) ,axis=-1)\nIMG2 = np.stack((R, G, BY) ,axis=-1)\nIMG3 = np.stack((RY, G, B) ,axis=-1)\nIMG4 = np.stack((R, GY, B) ,axis=-1)\n#IMG = np.divide(IMG, 255)\nIMG = cv2.resize(IMG,(299,299))\n\nfig2, ax2 = plt.subplots(2,2)\nfig2.set_size_inches(12,12)\nax2[0,0].set_title('R,G,B')\nax2[0,0].imshow(IMG)\nax2[0,1].set_title('R,G,BY')\nax2[0,1].imshow(IMG2)\nax2[1,0].set_title('RY,G,B')\nax2[1,0].imshow(IMG3)\nax2[1,1].set_title('R,GY,B')\nax2[1,1].imshow(IMG4)\nIMG.shape","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a325b86e947feff1b555f609006a3b9e62dbb42f"},"cell_type":"markdown","source":"## Define data-generator\ncredit:https://www.kaggle.com/byrachonok/pretrained-inceptionresnetv2-base-classifier"},{"metadata":{"_uuid":"a82e6cf7c3f069750bf23fe6eb84bce1aa408767"},"cell_type":"markdown","source":"In the cell below, I use PIL to do the augmentation for images and define a generator to generate training datas for keras model."},{"metadata":{"trusted":true,"_uuid":"577ddc5fec56b643a376e5022ec4d46b34855275"},"cell_type":"code","source":"#Define data_generator\n\nclass data_generator:\n    \n    def __init__(self):\n        pass\n    \n    def batch_train(self, idx, batch_size, shape, augment=True):\n        #extract eandom name and corresponding label\n        while True:\n            name_list = []\n            label_list = []\n\n            for n in np.random.choice(idx, batch_size):\n                name_list.append(n['name'])\n                int_label = list(map(int, n['label']))\n                label_list.append(int_label)\n\n            #batch_images = 提取images存成array, shape=(batch_size, shpae[0], shape[1], shpae[2]) = batch_images\n            batch_images = np.zeros((batch_size, shape[0], shape[1], shape[2]))\n            i = 0\n            for name in name_list:\n                image = self.load_img(name, shape)\n                if augment:\n                    image = self.augment(image)\n                batch_images[i] = image\n                i+=1\n\n            #batch_labels = 提取labels轉換為multiple one-hot, shape=(batch_size, 28)\n            batch_labels = np.zeros((batch_size, 28))\n            j = 0\n            for label in label_list:\n                batch_labels[j][label] = 1\n                j+=1\n\n            yield batch_images, batch_labels\n        \n    def load_img(self, name, shape):\n        R = np.array(Image.open(PATH_TRAIN+name+'_red.png'))\n        G = np.array(Image.open(PATH_TRAIN+name+'_green.png'))\n        B = np.array(Image.open(PATH_TRAIN+name+'_blue.png'))\n        Y = np.array(Image.open(PATH_TRAIN+name+'_yellow.png'))\n        image = np.stack((R, G, B, Y) ,axis=-1)\n        image = cv2.resize(image, (shape[0], shape[1]))\n        image = np.divide(image, 255)\n        return image\n    \n    def augment(self, image):\n        aug = iaa.OneOf([\n            iaa.Affine(rotate=90),\n            iaa.Affine(rotate=180),\n            iaa.Affine(rotate=270),\n            iaa.Fliplr(0.5),\n            iaa.Flipud(0.5)\n        ])\n        image = aug.augment_image(image)\n        return image","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"df7da8b8c9423fc69590c9cffb4969126478b4ea"},"cell_type":"markdown","source":"## Build model"},{"metadata":{"trusted":true,"_uuid":"7db3cb85c6cd9047b9a0cef2e59694d688b44989"},"cell_type":"code","source":"from keras import applications\nfrom keras.models import Model, Sequential, load_model\nfrom keras.layers import Input, Dense, Conv2D, MaxPooling2D, Activation, Flatten, Dropout, BatchNormalization\nfrom keras.callbacks import ModelCheckpoint\nfrom keras.optimizers import Adam, SGD\nimport tensorflow as tf\nimport keras.backend as K\nK.clear_session()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fb7ed72665548efa4091c581cb90d3c083d57b21"},"cell_type":"markdown","source":"In this cell I put loss fuctions, each are tried to test for training performance. Also, some utils are put over here.\n\nHere comes an important piece of my kernel: Focal loss. \nThis paper \"Focal Loss for Dense Object Detection\" https://arxiv.org/abs/1708.02002  introduce this loss fuction. This is originally designed for solving one-stage object detection accuracy problem. This paper claims that the major problem for one-stage object detection is the overwhelming imbalance between useful foreground(also hard and rare) labels and background(easy and much more) labels. They use focal loss (basically a modified cross_entropy loss) to make model focus on more difficult task(by fuctional high loss value) and not put too much attension on easy task. This is a perfect match for this competetion, in my examination focal loss really did works better than cross_entropy or f1_loss.\ncredit:\nhttps://www.kaggle.com/rejpalcz/best-loss-function-for-f1-score-metric\nhttps://blog.csdn.net/zziahgf/article/details/83589973"},{"metadata":{"trusted":true,"_uuid":"5cdad8d0b3a3cb8579069fa95e10e1b5e72e913c"},"cell_type":"code","source":"SHAPE = (299,299,4)\nBATCH_SIZE = 24\n\ndef f1(y_true, y_pred):\n    tp = K.sum(K.cast(y_true*y_pred, 'float'), axis=0)\n    fp = K.sum(K.cast((1-y_true)*y_pred, 'float'), axis=0)\n    fn = K.sum(K.cast(y_true*(1-y_pred), 'float'), axis=0)\n\n    p = tp / (tp + fp + K.epsilon())\n    r = tp / (tp + fn + K.epsilon())\n\n    f1 = 2*p*r / (p+r+K.epsilon())\n    f1 = tf.where(tf.is_nan(f1), tf.zeros_like(f1), f1)\n    return K.mean(f1)\n\ndef show_history(history):\n    fig, ax = plt.subplots(1, 3, figsize=(15,5))\n    ax[0].set_title('loss')\n    ax[0].plot(history.epoch, history.history[\"loss\"], label=\"Train loss\")\n    ax[0].plot(history.epoch, history.history[\"val_loss\"], label=\"Validation loss\")\n    ax[1].set_title('f1')\n    ax[1].plot(history.epoch, history.history[\"f1\"], label=\"Train f1\")\n    ax[1].plot(history.epoch, history.history[\"val_f1\"], label=\"Validation f1\")\n    ax[2].set_title('acc')\n    ax[2].plot(history.epoch, history.history[\"acc\"], label=\"Train acc\")\n    ax[2].plot(history.epoch, history.history[\"val_acc\"], label=\"Validation acc\")\n    ax[0].legend()\n    ax[1].legend()\n    ax[2].legend()\n    \ndef f1_loss(y_true, y_pred):\n    K_epsilon = K.epsilon()\n    #y_pred = K.cast(K.greater(K.clip(y_pred, 0, 1), THRESHOLD), K.floatx())\n    tp = K.sum(K.cast(y_true*y_pred, 'float'), axis=0)\n    tn = K.sum(K.cast((1-y_true)*(1-y_pred), 'float'), axis=0)\n    fp = K.sum(K.cast((1-y_true)*y_pred, 'float'), axis=0)\n    fn = K.sum(K.cast(y_true*(1-y_pred), 'float'), axis=0)\n\n    p = tp / (tp + fp + K_epsilon)\n    r = tp / (tp + fn + K_epsilon)\n\n    f1 = 2*p*r / (p+r+K_epsilon)\n    f1 = tf.where(tf.is_nan(f1), tf.zeros_like(f1), f1)\n    return 1-K.mean(f1)\n\ndef binary_focal_loss(y_true, y_pred, gamma=2.0, alpha=0.25):\n    \"\"\"\n    Implementation of Focal Loss from the paper in multiclass classification\n    Formula:\n        loss = -alpha_t*((1-p_t)^gamma)*log(p_t)\n        \n        p_t = y_pred, if y_true = 1\n        p_t = 1-y_pred, otherwise\n        \n        alpha_t = alpha, if y_true=1\n        alpha_t = 1-alpha, otherwise\n        \n        cross_entropy = -log(p_t)\n    Parameters:\n        alpha -- the same as wighting factor in balanced cross entropy\n        gamma -- focusing parameter for modulating factor (1-p)\n    Default value:\n        gamma -- 2.0 as mentioned in the paper\n        alpha -- 0.25 as mentioned in the paper\n    \"\"\"\n\n    # Define epsilon so that the backpropagation will not result in NaN\n    # for 0 divisor case\n    epsilon = K.epsilon()\n    # Add the epsilon to prediction value\n    #y_pred = y_pred + epsilon\n    # Clip the prediciton value\n    y_pred = K.clip(y_pred, epsilon, 1.0-epsilon)\n    # Calculate p_t\n    p_t = tf.where(K.equal(y_true, 1), y_pred, 1-y_pred)\n    # Calculate alpha_t\n    alpha_factor = K.ones_like(y_true)*alpha\n    alpha_t = tf.where(K.equal(y_true, 1), alpha_factor, 1-alpha_factor)\n    # Calculate cross entropy\n    cross_entropy = -K.log(p_t)\n    weight = alpha_t * K.pow((1-p_t), gamma)\n    # Calculate focal loss\n    loss = weight * cross_entropy\n    # Sum the losses in mini_batch\n    loss = K.sum(loss, axis=1)\n    \n    return loss\n\ndef SortedDict(adict): \n    new_dict = {}\n    ks = adict.keys() \n    ks = sorted(ks)\n    for key in ks:\n        new_dict[key] = adict[key]\n    return new_dict","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"85510af3b1cf3ee95ad1fdf870977b1c05b7cea8"},"cell_type":"code","source":"# As for my examine, this focal loss should works but I'm not 100% sure, so i did a small tensor graph\n# for checkng the value between function and my own math calculation.\ny_true = Input(shape=(None,))\ny_pred = Input(shape=(None,))\nloss_function = K.Function(inputs=[y_true,y_pred], outputs=[binary_focal_loss(y_true, y_pred)])\nmath_loss = -0.75*math.pow((1-0.6), 2)*math.log(0.6) - 0.25*math.pow((1-0.1), 2)*math.log(0.1)\nprint('By manual calculate focal_loss: ', math_loss)\ntensor_loss = loss_function([[[1,0,0,0,1]],[[1,0,0,0.4,0.1]]])\nprint('By tensor input via binary_focal loss', tensor_loss[0][0])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1cc80a18f4ea84970e95c1c861714bf9974db083"},"cell_type":"markdown","source":"Use the cell below to build my model. \nSince this is not the first round I train my model, I just put the original code below and use another cell to load my previous model and weights later."},{"metadata":{"trusted":true,"_uuid":"dd10030b4e78f0ef62a89d5612f76fa569cbbae4"},"cell_type":"code","source":"\"\"\"\n# load base model\nINPUT_SHAPE = (299,299,3)\nbase_model = applications.InceptionResNetV2(include_top=False ,weights='imagenet', input_shape=INPUT_SHAPE)\n\n# Add top-model to base_model\ndef make_classifier_model(input_dim=(8,8,1536)):\n    inp = Input(shape=input_dim)\n    X = Conv2D(128, kernel_size=(3,3), activation='relu')(inp)\n    X = MaxPooling2D(pool_size=(2, 2))(X)\n    X = BatchNormalization()(X)\n    X = Dropout(0.25)(X)\n    X = Conv2D(64, kernel_size=(1,1), activation='relu')(X)\n    X = BatchNormalization()(X)\n    X = Flatten()(X)  # this converts our 3D feature maps to 1D feature vectors\n    X = Dense(512, activation='relu')(X)\n    X = BatchNormalization()(X)\n    X = Dropout(0.5)(X)\n    X = Dense(256, activation='relu')(X)\n    X = BatchNormalization()(X)\n    X = Dropout(0.5)(X)\n    X = Dense(28)(X)\n    pred = Activation('sigmoid')(X)\n    classifier_model = Model(inp, pred, name='classifier_model')\n    return classifier_model\n\n# Add 4-channdel input layers to base_model\ndef make_input_model(shape=SHAPE):\n    inp = Input(shape=shape, name='input0')\n    pred = Conv2D(3,kernel_size=1,strides=1,padding='same',activation='tanh',\n                  kernel_regularizer=regularizers.l2(1e-4))(inp)\n    input_model = Model(inp, pred, name='input_model')\n    return input_model\n\n# Create model piece\nclassifier_model = make_classifier_model()\ninput_model = make_input_model()\n\n# Combine models\ninp = Input(shape=SHAPE, name='inputs')\nX = input_model(inp)\nX = base_model(X)\npred = classifier_model(X)\nmodel = Model(inp, pred, name='full_model')\n\nmodel.summary()\n\"\"\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"65db5a6f445fa36eec8eefe94fa8ce873a462eca"},"cell_type":"code","source":"#Use this cell to read model & weight\nmodel = load_model(PATH_BASE+MODEL_BASE+'fine_tune_weights.hdf5', custom_objects={'f1':f1, 'binary_focal_loss': binary_focal_loss})","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"88dd5f760d6d7e69a94e5cb228fe870c5cce4809"},"cell_type":"markdown","source":"## Train"},{"metadata":{"trusted":true,"_uuid":"cc39b992da683db9b7733d0bb5f92d147f4ee09e"},"cell_type":"code","source":"for layer in model.layers[:]:\n    layer.trainable =True\n\nmodel.compile(\n    loss=[binary_focal_loss],  \n    optimizer=Adam(1e-4),\n    metrics=['acc', f1])\n\n#verbose = 2 for every epoch output log\ncheckpointer = ModelCheckpoint('fine_tune_weights.hdf5', verbose=2, monitor='val_loss', save_best_only=True, mode='max')\n\ngenerator = data_generator()\ntrain_generator = generator.batch_train(train_idx, BATCH_SIZE, SHAPE, augment=True)\nvalidation_generator = generator.batch_train(test_idx, 620, SHAPE, augment=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ce870ce7f1020d648050e2948c10174400de95e0"},"cell_type":"code","source":"history = model.fit_generator(\n    train_generator,\n    steps_per_epoch=100,\n    validation_data=next(validation_generator),\n    epochs=80,\n    verbose=1,\n    callbacks=[checkpointer])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8cbf6bfd79e95c7dbed635db3654396817a0c747"},"cell_type":"code","source":"show_history(history)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b1256dab6121e1459d2800070b24f97d1311f55a"},"cell_type":"markdown","source":"## Choose threshold value base on the F1 score"},{"metadata":{"trusted":true,"_uuid":"0ad013cc2013f9cd78f70d7378a1c4e555da6bfc"},"cell_type":"code","source":"n_list = np.arange(0.1,0.5,0.02)\nfor idx in test_idx[:3]:\n    name0 = idx['name']\n    print(idx)\n    print(idx['name'])\n    print(name0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f4cbad5272fba0113d8fed4d7eee0bc70d15d859"},"cell_type":"code","source":"%%time\nfrom tqdm import tqdm\nTP_data = {}\nFP_data = {}\nFN_data = {}\nF1_best = 0\nF1_ther = 0\nfor threshold in tqdm(n_list):\n    F1_sum = 0\n    TP_datai = {}\n    FP_datai = {}\n    FN_datai = {}\n    for i in range(28):\n        TP_datai[i] = 0\n        FP_datai[i] = 0\n        FN_datai[i] = 0\n    for idx in test_idx[:500]:\n        name0 = idx['name']\n        generator = data_generator()\n        image = generator.load_img(name0, SHAPE)\n        score_predict = model.predict(image[np.newaxis,:])\n        score_predict = np.array(score_predict)[0]\n        label_predict = np.arange(28)[score_predict>=threshold]\n        true_label = idx['label']\n        true_label = np.array(true_label).astype(int)\n        label_predict = set(label_predict)\n        true_label = set(true_label)\n#         print(label_predict,'label predict')\n#         print(true_label,'true_label')\n        \n        TP = sum(1 for num in label_predict if num in true_label)\n#         print(TP,'TP')\n        FP = sum(1 for num in label_predict if not num in true_label)\n#         print(FP,'FP')\n        FN = sum(1 for num in true_label if not num in label_predict)\n#         print(FN,'FN')\n        TN = 28 - (TP+FP+FN)\n        F1_sum += 2*TP/(2*TP+FN+FP)\n        \n        # count for acc for every label type\n#         TP_count = 0\n#         FP_count = 0\n#         FN_count = 0\n        for num in label_predict:\n            if num in true_label:\n#                 TP_count+=1\n                TP_datai[num] += 1\n            if num not in true_label:\n#                 FP_count+=1\n                FP_datai[num] += 1\n        for num in true_label:\n            if num not in label_predict:\n#                 FN_count+=1\n                FN_datai[num] += 1\n        \n        \n    if F1_sum>F1_best:\n        F1_best = F1_sum\n        F1_thre = threshold\n        TP_data = TP_datai\n        FP_data = FP_datai\n        FN_data = FN_datai\n        \n    print('F1_score_sum: ', F1_sum, 'at threshold: ', threshold)\nTP_data = SortedDict(TP_data)\nFP_data = SortedDict(FP_data)\nFN_data = SortedDict(FN_data)\nprint('F1_best ', F1_best, '  F1_thre ', F1_thre)\nprint('TP_data ', TP_data)\nprint('FP_data ', FP_data)\nprint('FN_data ', FN_data)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"be74fe135c88db64d4ae1c1729aff9c9c10ef243"},"cell_type":"code","source":"def dict_to_barh(dict_data, title):\n    x = list(dict_data.keys())\n    y = list(dict_data.values())\n    return plt_barh(x, y, title)\n\ndict_to_barh(TP_data, 'TP_data')\ndict_to_barh(FP_data, 'FP_data')\ndict_to_barh(FN_data, 'FN_data')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e1e24eb3f40c822d5ebe025847a48f038dc57265"},"cell_type":"markdown","source":"## Submit"},{"metadata":{"trusted":true,"_uuid":"a8e2e53029194871ea7cccaaa989e8ca0b56de88"},"cell_type":"code","source":"submit = pd.read_csv(PATH_BASE+TRAIN_BASE+'sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7358b96b8d14c59b2e40cf0e3911ee3b14d35bbb"},"cell_type":"code","source":"%%time\nPATH_TRAIN = PATH_BASE+TRAIN_BASE+'test/'\ngenerator = data_generator()\npredicted = []\n\nfor name in tqdm(submit['Id']):\n    #path = os.path.join('../input/test/', name)\n    image = generator.load_img(name, SHAPE)\n    score_predict = model.predict(image[np.newaxis,:])[0]\n    label_predict = np.arange(28)[score_predict>=F1_thre]\n    str_predict_label = ' '.join(str(l) for l in label_predict)\n    predicted.append(str_predict_label)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c623427a25b51a39e774739f057c9b7cf7febae6"},"cell_type":"code","source":"submit['Predicted'] = predicted\nsubmit.to_csv('4 channel V2 with rare T2 plus threshold.csv', index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}