{"cells":[{"metadata":{"_uuid":"d2fd95afc524740e3d1d750e6645706ba089bd75"},"cell_type":"markdown","source":"## Intro\nthese are not typical RGB images, each of them is seperated into RGBY images. Also, they are not only color itself, each color infers different structures in human cells. Lastly, this is a multi-class + multi-label mission, and there exist strong imbalance between all 28 types of protein(labels). Some labels are extremely rare while some are much mor frequently labeled.\n\nHere is my kernel for dealing with all these issues, as a record for my first DL problem."},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"import os\nimport numpy as np\nimport pandas as pd\nfrom PIL import Image\nfrom zipfile import ZipFile\nimport matplotlib.pyplot as plt\nimport cv2\nimport math\nfrom imgaug import augmenters as iaa\n%matplotlib inline\n\nimport time\nt_start = time.time()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ee04dc60d6f582f796b568461aa58c8aca9dd351"},"cell_type":"code","source":"os.listdir('../input')\nPATH_BASE = '../input/'\nPATH_TRAIN = PATH_BASE+'train/'\n\nSHAPE = (299,299,4)\nBATCH_SIZE = 30\nEPOCHS = 10","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7ac7e4c4a6e8f87c37c5bb9f27c537fa25979b54"},"cell_type":"code","source":"raw_labels = pd.read_csv(PATH_BASE+'train.csv')\ndata_names = os.listdir(PATH_TRAIN)\n\n#extract label names and labels array[{name: ,label:}]\nlabels = []\nfor name, label in zip(raw_labels['Id'],raw_labels['Target'].str.split(\" \")):\n    labels.append({\n        'name':name,\n        'label':label\n    })\n    \n#Split data to train/dev set\nfrom sklearn.model_selection import train_test_split\ntrain_idx, test_idx = train_test_split(labels, test_size=0.2)\nprint('train: ' + str(len(train_idx)) + '\\n'+ 'validation: ' + str(len(test_idx)))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f31228d03eb295ce84a3671f39904e1a22f849cf"},"cell_type":"markdown","source":" # Data Visualization\n credit:\n https://www.kaggle.com/wordroid/resnet50-4x256-globalmax-lb-0-443\n"},{"metadata":{"_uuid":"6b9664c26d0df64da8f06c61dc971d51ec2c9d51"},"cell_type":"markdown","source":"In the cell below, we can observe data imbalance. \nWhile label 0 and 25 occupy 67% of the training set, label 8,9,10,15,27 only take approximately 0.001%."},{"metadata":{"trusted":true,"_uuid":"83677e2729802a26af60ada060c0a194b80b6269"},"cell_type":"code","source":"y_cat_train_dic = {}\nfor icat in range(28):\n    target = str(icat)\n    y_cat_train_5 = np.array([int(target in ee['label']) for ee in train_idx])\n    y_cat_train_dic[icat] = y_cat_train_5\nup_sample = {}\nfor k in y_cat_train_dic:\n    v = y_cat_train_dic[k].sum()\n    up_sample[k] = np.round(v / len(train_idx), 5)\nprint(up_sample)\ndef plt_barh(x, y, title):\n    fig, ax = plt.subplots(figsize=(15,7))\n    width = 0.75\n    ind = np.arange(len(up_sample))  # the x locations for the groups\n    ax.barh(ind, y, width, color=\"blue\")\n    ax.set_yticks(ind+width/2)\n    ax.set_yticklabels(x, minor=False)\n    plt.title(title)\n    for i, v in enumerate(y):\n        ax.text(v, i , str(v), color='blue', fontweight='bold')\n    plt.xlabel('x')\n    plt.ylabel('y')\nx = list(up_sample.keys())\ny = list(up_sample.values())\nplt_barh(x, y, 'data imbalance')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"91a222e94f7ddc697d2ccc9a325a91911378d3da"},"cell_type":"markdown","source":"In the cell below, four channels of images are showd. \nAlso, I tried to merge Y channel into RGB in different ways for my training set.\n\nNote: Afterwards, I chose to use 4-channels for my model. Actually, merging Y channel to train a 3-channels model also works, got about 0.4 LB score. However, since useing 3 or 4 channels doesn't show significant difference in training performance(in my case), I chose 4-channel because it's more reasonable to use as much as the informations we can access in datas."},{"metadata":{"trusted":true,"_uuid":"514f705937efc603aac2a3298151587dda758c7f"},"cell_type":"code","source":"test = labels[10]\nprint(test); print(test['name']); print(test['label'])\n\nfig, ax = plt.subplots(1,4,figsize=(12,12))\nfig.tight_layout()\n\n#Try different mix method\nnames = [n['name'] for n in np.random.choice(labels, 1)]\nR = np.array(Image.open(PATH_TRAIN+names[0]+'_red.png'))\nax[0].imshow(R,cmap='Reds')\nax[0].set_title('R')\nG = np.array(Image.open(PATH_TRAIN+names[0]+'_green.png'))\nax[1].imshow(G,cmap='Greens')\nax[1].set_title('G')\nB = np.array(Image.open(PATH_TRAIN+names[0]+'_blue.png'))\nax[2].imshow(B,cmap='Blues')\nax[2].set_title('B')\nY = np.array(Image.open(PATH_TRAIN+names[0]+'_yellow.png'))\nax[3].imshow(Y,cmap='YlOrBr')\nax[3].set_title('Y')\n\nBY = (B+Y)\nBY[BY>255] = 255\nRY = (R+Y)\nRY[RY>255] = 255\nGY = (G+Y)\nGY[GY>255] = 255\n\nIMG = np.stack((R, G, B) ,axis=-1)\nIMG2 = np.stack((R, G, BY) ,axis=-1)\nIMG3 = np.stack((RY, G, B) ,axis=-1)\nIMG4 = np.stack((R, GY, B) ,axis=-1)\nIMG = cv2.resize(IMG,(299,299))\n\nfig2, ax2 = plt.subplots(2,2)\nfig2.set_size_inches(12,12)\nax2[0,0].set_title('R,G,B')\nax2[0,0].imshow(IMG)\nax2[0,1].set_title('R,G,BY')\nax2[0,1].imshow(IMG2)\nax2[1,0].set_title('RY,G,B')\nax2[1,0].imshow(IMG3)\nax2[1,1].set_title('R,GY,B')\nax2[1,1].imshow(IMG4)\nIMG.shape","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a325b86e947feff1b555f609006a3b9e62dbb42f"},"cell_type":"markdown","source":"## Define data-generator\ncredit:https://www.kaggle.com/byrachonok/pretrained-inceptionresnetv2-base-classifier"},{"metadata":{"_uuid":"a82e6cf7c3f069750bf23fe6eb84bce1aa408767"},"cell_type":"markdown","source":"In the cell below, I use PIL to do the augmentation for images and define a generator to generate training datas for keras model."},{"metadata":{"trusted":true,"_uuid":"577ddc5fec56b643a376e5022ec4d46b34855275"},"cell_type":"code","source":"#Define data_generator\nclass data_generator:\n    \n    def __init__(self):\n        pass\n    \n    def batch_train(self, idx, batch_size, shape, augment=True):\n        #extract eandom name and corresponding label\n        while True:\n            name_list = []\n            label_list = []\n\n            for n in np.random.choice(idx, batch_size):\n                name_list.append(n['name'])\n                int_label = list(map(int, n['label']))\n                label_list.append(int_label)\n\n            #batch_images = 提取images存成array, shape=(batch_size, shpae[0], shape[1], shpae[2]) = batch_images\n            batch_images = np.zeros((batch_size, shape[0], shape[1], shape[2]))\n            i = 0\n            for name in name_list:\n                image = self.load_img(name, shape)\n                if augment:\n                    image = self.augment(image)\n                batch_images[i] = image\n                i+=1\n\n            #batch_labels = 提取labels轉換為multiple one-hot, shape=(batch_size, 28)\n            batch_labels = np.zeros((batch_size, 28))\n            j = 0\n            for label in label_list:\n                batch_labels[j][label] = 1\n                j+=1\n\n            yield batch_images, batch_labels\n        \n    def load_img(self, name, shape):\n        R = np.array(Image.open(PATH_TRAIN+name+'_red.png'))\n        G = np.array(Image.open(PATH_TRAIN+name+'_green.png'))\n        B = np.array(Image.open(PATH_TRAIN+name+'_blue.png'))\n        Y = np.array(Image.open(PATH_TRAIN+name+'_yellow.png'))\n        image = np.stack((R, G, B, Y) ,axis=-1)\n        image = cv2.resize(image, (shape[0], shape[1]))\n        image = np.divide(image, 255)\n        return image\n    \n    def augment(self, image):\n        aug = iaa.OneOf([\n            iaa.Affine(rotate=90),\n            iaa.Affine(rotate=180),\n            iaa.Affine(rotate=270),\n            iaa.Fliplr(0.5),\n            iaa.Flipud(0.5)\n        ])\n        image = aug.augment_image(image)\n        return image","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"df7da8b8c9423fc69590c9cffb4969126478b4ea"},"cell_type":"markdown","source":"## Build model"},{"metadata":{"trusted":true,"_uuid":"7db3cb85c6cd9047b9a0cef2e59694d688b44989"},"cell_type":"code","source":"from keras import applications\nfrom keras.models import Model, Sequential, load_model\nfrom keras.layers import Input, Dense, Conv2D, MaxPooling2D, Activation, Flatten, Dropout, BatchNormalization\nfrom keras.callbacks import ModelCheckpoint\nfrom keras.optimizers import Adam, SGD\nfrom keras import regularizers\nimport tensorflow as tf\nimport keras.backend as K\nK.clear_session()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fb7ed72665548efa4091c581cb90d3c083d57b21"},"cell_type":"markdown","source":"In this cell I put loss fuctions, each are tried to test for training performance. Also, some utils are put over here.\n\nHere comes an important piece of my kernel: Focal loss. \nThis paper \"Focal Loss for Dense Object Detection\" https://arxiv.org/abs/1708.02002  introduce this loss fuction. This is originally designed for solving one-stage object detection accuracy problem. This paper claims that the major problem for one-stage object detection is the overwhelming imbalance between useful foreground(also hard and rare) labels and background(easy and much more) labels. They use focal loss (basically a modified cross_entropy loss) to make model focus on more difficult task(by fuctional high loss value) and not put too much attension on easy task. This is a perfect match for this competetion, in my examination focal loss really did works better than cross_entropy or f1_loss.\ncredit:\nhttps://www.kaggle.com/rejpalcz/best-loss-function-for-f1-score-metric\nhttps://blog.csdn.net/zziahgf/article/details/83589973"},{"metadata":{"trusted":true,"_uuid":"5cdad8d0b3a3cb8579069fa95e10e1b5e72e913c"},"cell_type":"code","source":"THRESHOLD = 0.5\n\nK_epsilon = K.epsilon()\ndef f1(y_true, y_pred):\n    y_pred = K.cast(K.greater(K.clip(y_pred, 0, 1), THRESHOLD), K.floatx())\n    tp = K.sum(K.cast(y_true*y_pred, 'float'), axis=0)\n    tn = K.sum(K.cast((1-y_true)*(1-y_pred), 'float'), axis=0)\n    fp = K.sum(K.cast((1-y_true)*y_pred, 'float'), axis=0)\n    fn = K.sum(K.cast(y_true*(1-y_pred), 'float'), axis=0)\n\n    p = tp / (tp + fp + K_epsilon)\n    r = tp / (tp + fn + K_epsilon)\n\n    f1 = 2*p*r / (p+r+K_epsilon)\n    f1 = tf.where(tf.is_nan(f1), tf.zeros_like(f1), f1)\n    return K.mean(f1)\n\ndef f1_loss(y_true, y_pred):\n    tp = K.sum(K.cast(y_true*y_pred, 'float'), axis=0)\n    tn = K.sum(K.cast((1-y_true)*(1-y_pred), 'float'), axis=0)\n    fp = K.sum(K.cast((1-y_true)*y_pred, 'float'), axis=0)\n    fn = K.sum(K.cast(y_true*(1-y_pred), 'float'), axis=0)\n\n    p = tp / (tp + fp + K_epsilon)\n    r = tp / (tp + fn + K_epsilon)\n\n    f1 = 2*p*r / (p+r+K_epsilon)\n    f1 = tf.where(tf.is_nan(f1), tf.zeros_like(f1), f1)\n    return 1-K.mean(f1)\n\ndef show_history(history):\n    fig, ax = plt.subplots(1, 3, figsize=(15,5))\n    ax[0].set_title('loss')\n    ax[0].plot(history.epoch, history.history[\"loss\"], label=\"Train loss\")\n    ax[0].plot(history.epoch, history.history[\"val_loss\"], label=\"Validation loss\")\n    ax[1].set_title('f1')\n    ax[1].plot(history.epoch, history.history[\"f1\"], label=\"Train f1\")\n    ax[1].plot(history.epoch, history.history[\"val_f1\"], label=\"Validation f1\")\n    ax[2].set_title('categorical_accuracy')\n    ax[2].plot(history.epoch, history.history[\"categorical_accuracy\"], label=\"Train categorical_accuracy\")\n    ax[2].plot(history.epoch, history.history[\"val_categorical_accuracy\"], label=\"Validation categorical_accuracy\")\n    ax[0].legend()\n    ax[1].legend()\n    ax[2].legend()\n\ndef SortedDict(adict): \n    new_dict = {}\n    ks = adict.keys() \n    ks = sorted(ks)\n    for key in ks:\n        new_dict[key] = adict[key]\n    return new_dict","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"dd10030b4e78f0ef62a89d5612f76fa569cbbae4"},"cell_type":"code","source":"# load base model\nINPUT_SHAPE = (299,299,3)\nbase_model = applications.InceptionResNetV2(include_top=False ,weights='imagenet', input_shape=INPUT_SHAPE)\n\nfor l in base_model.layers[::-1][:]: # enable training just .. Layers\n    l.trainable = True\n\n# Add top-model to base_model\ndef make_classifier_model(input_dim=(8,8,1536)):\n    inp = Input(shape=input_dim)\n    X = Conv2D(128, kernel_size=(3,3), activation='relu')(inp)\n    X = MaxPooling2D(pool_size=(2, 2))(X)\n    X = BatchNormalization()(X)\n    X = Dropout(0.25)(X)\n    X = Conv2D(64, kernel_size=(1,1), activation='relu')(X)\n    X = BatchNormalization()(X)\n    X = Flatten()(X)  # this converts our 3D feature maps to 1D feature vectors\n    X = Dense(512, activation='relu')(X)\n    X = BatchNormalization()(X)\n    X = Dropout(0.5)(X)\n    X = Dense(256, activation='relu')(X)\n    X = BatchNormalization()(X)\n    X = Dropout(0.5)(X)\n    X = Dense(28)(X)\n    pred = Activation('sigmoid')(X)\n    classifier_model = Model(inp, pred, name='classifier_model')\n    return classifier_model\n\n# Add 4-channdel input layers to base_model\ndef make_input_model(shape=SHAPE):\n    inp = Input(shape=shape, name='input0')\n    pred = Conv2D(3,kernel_size=1,strides=1,padding='same',activation='tanh',\n                  kernel_regularizer=regularizers.l2(1e-4))(inp)\n    input_model = Model(inp, pred, name='input_model')\n    return input_model\n\n# Create model piece\nclassifier_model = make_classifier_model()\ninput_model = make_input_model()\n\n# Combine models\ninp = Input(shape=SHAPE, name='inputs')\nX = input_model(inp)\nX = base_model(X)\npred = classifier_model(X)\nmodel = Model(inp, pred, name='full_model')\n\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"20a3e8b272a0d73474234c8260ad919fc487f3f2"},"cell_type":"code","source":"#!/usr/bin/env python3\n# -*- coding: utf-8 -*-\n\"\"\"\nStochastic Weight Averaging: https://arxiv.org/abs/1803.05407\nImplementaton in Keras from user defined epochs assuming constant \nlearning rate\nCyclic learning rate implementation in https://arxiv.org/abs/1803.05407 \nnot implemented\nCreated on July 4, 2018\n@author: Krist Papadopoulos\n\"\"\"\n\nimport keras\n\nclass SWA(keras.callbacks.Callback):\n    \n    def __init__(self, filepath, swa_epoch):\n        super(SWA, self).__init__()\n        self.filepath = filepath\n        self.swa_epoch = swa_epoch \n    \n    def on_train_begin(self, logs=None):\n        self.nb_epoch = self.params['epochs']\n        print('Stochastic weight averaging selected for last {} epochs.'\n              .format(self.nb_epoch - self.swa_epoch))\n        \n    def on_epoch_end(self, epoch, logs=None):\n        \n        if epoch == self.swa_epoch:\n            self.swa_weights = self.model.get_weights()\n            \n        elif epoch > self.swa_epoch:    \n            for i, layer in enumerate(self.model.layers):\n                self.swa_weights[i] = (self.swa_weights[i] * \\\n                    (epoch - self.swa_epoch) + self.model.get_weights()[i]) \\\n                    /((epoch - self.swa_epoch)  + 1)  \n\n        else:\n            pass\n        \n    def on_train_end(self, logs=None):\n        self.model.set_weights(self.swa_weights)\n        print('Final model parameters set to stochastic weight average.')\n        self.model.save_weights(self.filepath)\n        print('Final stochastic averaged weights saved to file.')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c3f78070630aa5d24e78a54df9145d3a18529d17"},"cell_type":"markdown","source":"## Fine-Tuning"},{"metadata":{"trusted":true,"_uuid":"6fe8a7e81ec57f78eff1f146ecd3b9a5da9cde1e"},"cell_type":"code","source":"from keras.callbacks import EarlyStopping, ReduceLROnPlateau, ModelCheckpoint\n\nBestModelWeightsPath = 'BestModel.hdf5'\nswa = SWA('Best_Weights.hdf5', int(EPOCHS * 0.9))\ncheck_point = ModelCheckpoint(\n    BestModelWeightsPath, monitor='val_f1', verbose=1,\n    save_best_only=True, \n    mode='max',\n)\nreduce_lr = ReduceLROnPlateau(monitor='val_f1', factor=0.5, min_delta=0.0001, patience=10, verbose=1)\nearlyStop = EarlyStopping(monitor='val_f1', mode='max', patience=30, verbose=1)\ncallbacks_list = [check_point, reduce_lr, earlyStop, swa]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cc39b992da683db9b7733d0bb5f92d147f4ee09e"},"cell_type":"code","source":"from keras.metrics import categorical_accuracy\n\nmodel.compile( loss=f1_loss, optimizer=Adam(1e-3), metrics=['categorical_accuracy', f1])\ngenerator = data_generator()\ntrain_generator = generator.batch_train(train_idx, BATCH_SIZE, SHAPE, augment=True)\nvalidation_generator = generator.batch_train(test_idx, 620, SHAPE, augment=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ce870ce7f1020d648050e2948c10174400de95e0"},"cell_type":"code","source":"history = model.fit_generator(\n    train_generator,\n    steps_per_epoch= len(train_idx) // BATCH_SIZE,\n    validation_data= next(validation_generator),\n    epochs=EPOCHS,\n    verbose=1,\n    callbacks=callbacks_list\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8cbf6bfd79e95c7dbed635db3654396817a0c747"},"cell_type":"code","source":"show_history(history)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"47d5780d7e60a7705b0d0e6d82baf51fac5a4376"},"cell_type":"code","source":"#Use this cell to read model & weight\nmodel.load_weights('Best_Weights.hdf5')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b1256dab6121e1459d2800070b24f97d1311f55a"},"cell_type":"markdown","source":"## Choose threshold value base on the F1 score"},{"metadata":{"trusted":true,"_uuid":"0ad013cc2013f9cd78f70d7378a1c4e555da6bfc"},"cell_type":"code","source":"n_list = np.arange(0.1,0.5,0.02)\nfor idx in test_idx[:3]:\n    name0 = idx['name']\n    print(idx)\n    print(idx['name'])\n    print(name0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f4cbad5272fba0113d8fed4d7eee0bc70d15d859","scrolled":false},"cell_type":"code","source":"%%time\nfrom tqdm import tqdm\nTP_data = {}\nFP_data = {}\nFN_data = {}\nF1_best = 0\nF1_ther = 0\nfor threshold in tqdm(n_list):\n    F1_sum = 0\n    TP_datai = {}\n    FP_datai = {}\n    FN_datai = {}\n    for i in range(28):\n        TP_datai[i] = 0\n        FP_datai[i] = 0\n        FN_datai[i] = 0\n    for idx in test_idx[:500]:\n        name0 = idx['name']\n        generator = data_generator()\n        image = generator.load_img(name0, SHAPE)\n        score_predict = model.predict(image[np.newaxis,:])\n        score_predict = np.array(score_predict)[0]\n        label_predict = np.arange(28)[score_predict>=threshold]\n        true_label = idx['label']\n        true_label = np.array(true_label).astype(int)\n        label_predict = set(label_predict)\n        true_label = set(true_label)\n        \n        TP = sum(1 for num in label_predict if num in true_label)\n        FP = sum(1 for num in label_predict if not num in true_label)\n        FN = sum(1 for num in true_label if not num in label_predict)\n        TN = 28 - (TP+FP+FN)\n        F1_sum += 2*TP/(2*TP+FN+FP)\n        \n        # count for acc for every label type\n        for num in label_predict:\n            if num in true_label:\n                TP_datai[num] += 1\n            if num not in true_label:\n                FP_datai[num] += 1\n        for num in true_label:\n            if num not in label_predict:\n                FN_datai[num] += 1\n        \n        \n    if F1_sum>F1_best:\n        F1_best = F1_sum\n        F1_thre = threshold\n        TP_data = TP_datai\n        FP_data = FP_datai\n        FN_data = FN_datai\n        \n    print('F1_score_sum: ', F1_sum, 'at threshold: ', threshold)\nTP_data = SortedDict(TP_data)\nFP_data = SortedDict(FP_data)\nFN_data = SortedDict(FN_data)\nprint('F1_best ', F1_best, '  F1_thre ', F1_thre)\nprint('TP_data ', TP_data)\nprint('FP_data ', FP_data)\nprint('FN_data ', FN_data)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"be74fe135c88db64d4ae1c1729aff9c9c10ef243"},"cell_type":"code","source":"def dict_to_barh(dict_data, title):\n    x = list(dict_data.keys())\n    y = list(dict_data.values())\n    return plt_barh(x, y, title)\n\ndict_to_barh(TP_data, 'TP_data')\ndict_to_barh(FP_data, 'FP_data')\ndict_to_barh(FN_data, 'FN_data')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e1e24eb3f40c822d5ebe025847a48f038dc57265"},"cell_type":"markdown","source":"## Submit"},{"metadata":{"trusted":true,"_uuid":"a8e2e53029194871ea7cccaaa989e8ca0b56de88"},"cell_type":"code","source":"submit = pd.read_csv(PATH_BASE+'sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7358b96b8d14c59b2e40cf0e3911ee3b14d35bbb"},"cell_type":"code","source":"%%time\nPATH_TRAIN = PATH_BASE+'test/'\ngenerator = data_generator()\npredicted = []\n\nfor name in tqdm(submit['Id']):\n    image = generator.load_img(name, SHAPE)\n    score_predict = model.predict(image[np.newaxis,:])[0]\n    label_predict = np.arange(28)[score_predict>=F1_thre]\n    str_predict_label = ' '.join(str(l) for l in label_predict)\n    predicted.append(str_predict_label)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c623427a25b51a39e774739f057c9b7cf7febae6"},"cell_type":"code","source":"submit['Predicted'] = predicted\nsubmit.to_csv('4 channel V2 with rare plus threshold.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b7c4b3f3934ceb3886aea899aad8fbc4622c0dc6"},"cell_type":"code","source":"t_finish = time.time()\nprint(f\"Kernel run time = {(t_finish-t_start)/3600} hours\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c7493f1a0345afc8dfb91223947d01c55e8c0bd5"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}