{"cells":[{"metadata":{"_uuid":"ed2430a20065902d7aa99b0c3265c4455758f182"},"cell_type":"markdown","source":"### Introduction\nLet's try to understand how to use channels in learning. First variant is combine them all together like [here](https://www.kaggle.com/byrachonok/pretrained-inceptionresnetv2-base-classifier) \n\nIn our case we will use two branch of network - first one is about data and second one is about \"The protein of interest\". We will get 2 images from 4 sources:\n1. Yellow, blue and red channel - we create RGB image (yellow channel will be in fact green color now);\n2. Source green channel will be grayscale image but with 3 equal channel (condition for using Imagenet weights)\n\nLet's start from imports and data loading:"},{"metadata":{"_uuid":"5b508697e471d7a20d9df8a087e1d9e2624f5922"},"cell_type":"markdown","source":"### Imports:"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os, sys\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport skimage.io\n\nfrom scipy.misc import imread, imresize\nfrom skimage.transform import resize\nfrom tqdm import tqdm\n\nimport keras\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.applications import Xception\nfrom keras.applications.inception_resnet_v2 import InceptionResNetV2\nfrom keras.models import Sequential, Model, load_model\nfrom keras.layers import Activation, Dense, Multiply, Input\nfrom keras.callbacks import ModelCheckpoint\nfrom keras import metrics\nfrom keras.optimizers import Adam  \nfrom keras import backend as K\n\nfrom itertools import chain\nfrom collections import Counter\nimport warnings\nwarnings.filterwarnings(\"ignore\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"72c425f3aaa2645215a3adc808fc188c5b59106d"},"cell_type":"markdown","source":"### Load dataset info:"},{"metadata":{"trusted":true,"_uuid":"42315d01b95b8a901088befa4100b014a1416c7a"},"cell_type":"code","source":"path_to_train = '../input/train/'\ndata = pd.read_csv('../input/train.csv')\n\ntrain_dataset_info = []\nfor name, labels in zip(data['Id'], data['Target'].str.split(' ')):\n    train_dataset_info.append({\n        'path':os.path.join(path_to_train, name),\n        'labels':np.array([int(label) for label in labels])})\ntrain_dataset_info = np.array(train_dataset_info)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ae3dcb3e4a000051fc8947502a28d399d200a924"},"cell_type":"markdown","source":"6### Create datagenerator\n\nOur generator will return next structure: ([RGB_images, grayscale_images], labels). As described above, now we will have two inputs"},{"metadata":{"trusted":true,"_uuid":"c21509d05e2882e6315fc7390d27658c0654fc15"},"cell_type":"code","source":"class DataGenerator:\n    def __init__(self):\n        self.image_generator = ImageDataGenerator(rescale=1. / 255,\n                                     vertical_flip=True,\n                                     horizontal_flip=True,\n                                     rotation_range=180,\n                                     fill_mode='reflect')\n    def create_train(self, dataset_info, batch_size, shape, augument=True):\n        assert shape[2] == 3\n        while True:\n            random_indexes = np.random.choice(len(dataset_info), batch_size)\n            batch_images1 = np.empty((batch_size, shape[0], shape[1], shape[2]))\n            batch_images2 = np.empty((batch_size, shape[0], shape[1], shape[2]))\n            batch_labels = np.zeros((batch_size, 28))\n            for i, idx in enumerate(random_indexes):\n                image1, image2 = self.load_image(\n                    dataset_info[idx]['path'], shape)\n                batch_images1[i] = image1\n                batch_images2[i] = image2\n                batch_labels[i][dataset_info[idx]['labels']] = 1\n            yield [batch_images1, batch_images2], batch_labels\n            \n    \n    def load_image(self, path, shape):\n        image_red_ch = skimage.io.imread(path+'_red.png')\n        image_yellow_ch = skimage.io.imread(path+'_yellow.png')\n        image_green_ch = skimage.io.imread(path+'_green.png')\n        image_blue_ch = skimage.io.imread(path+'_blue.png')\n\n        image1 = np.stack((\n            image_red_ch, \n            image_yellow_ch, \n            image_blue_ch), -1)\n        image2 = np.stack((\n            image_green_ch, \n            image_green_ch, \n            image_green_ch), -1)\n        image1 = resize(image1, (shape[0], shape[1], 3), mode='reflect')\n        image2 = resize(image2, (shape[0], shape[1], 3), mode='reflect')\n        return image1.astype(np.float), image2.astype(np.float)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"74687ae11e12c8d0559e5421a356a67e8ab01537"},"cell_type":"markdown","source":"### Show data:"},{"metadata":{"trusted":true,"_uuid":"3f2ae48955c4b75b13dd9e9ad0d9e1c84214b019"},"cell_type":"code","source":"# create train datagen\ntrain_datagen = DataGenerator()\n\ngenerator = train_datagen.create_train(\n    train_dataset_info, 5, (299,299,3))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"42b71a5e01941351187bc971f39486bd3c20836f"},"cell_type":"markdown","source":"### Visualization\nFirst line is RGB images, second line is relevant grayscale images:"},{"metadata":{"trusted":true,"_uuid":"5f261d54682cbd326a01e09a0f92598228145d2d","_kg_hide-input":false,"_kg_hide-output":false},"cell_type":"code","source":"images, labels = next(generator)\nimages1, images2 = images\nfig, ax = plt.subplots(2,5,figsize=(25,15))\nfor i in range(5):\n    ax[0, i].imshow(images1[i])\nfor i in range(5):\n    ax[1, i].imshow(images2[i])\nprint('min: {0}, max: {1}'.format(images1.min(), images1.max()))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"69132a0eb0f4691fbb6e20d7a119fa19616c77f8"},"cell_type":"markdown","source":"### Split data\nSplit data into train and val part with a ratio 80/20, we will use for it \"hack\" from [this kernel](https://www.kaggle.com/kmader/rgb-transfer-learning-with-inceptionv3-for-protein)"},{"metadata":{"trusted":true,"_uuid":"6b4e7446f2d6e07a793797d6c5662ed1d3bdc9e0"},"cell_type":"code","source":"# from https://www.kaggle.com/kmader/rgb-transfer-learning-with-inceptionv3-for-protein\ndata['target_list'] = data['Target'].map(lambda x: [int(a) for a in x.split(' ')])\nall_labels = list(chain.from_iterable(data['target_list'].values))\nc_val = Counter(all_labels)\nn_keys = c_val.keys()\nmax_idx = max(n_keys)\ndata['target_vec'] = data['target_list'].map(lambda ck: [i in ck for i in range(max_idx+1)])\nfrom sklearn.model_selection import train_test_split\ntrain_df, valid_df = train_test_split(data, \n                 test_size = 0.2, \n                  # hack to make stratification work                  \n                 stratify = data['Target'].map(lambda x: x[:3] if '27' not in x else '0'))\nprint(train_df.shape[0], 'training masks')\nprint(valid_df.shape[0], 'validation masks')\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c04a3357145920003e2c546a7666710e67d731ae"},"cell_type":"markdown","source":"### Create lists for training:"},{"metadata":{"trusted":true,"_uuid":"6f4fbc75c45e49ab6437e685c4f0163a634ebcd9"},"cell_type":"code","source":"train_dataset_info = []\nfor name, labels in zip(train_df['Id'], train_df['Target'].str.split(' ')):\n    train_dataset_info.append({\n        'path':os.path.join(path_to_train, name),\n        'labels':np.array([int(label) for label in labels])})\ntrain_dataset_info = np.array(train_dataset_info)\nvalid_dataset_info = []\nfor name, labels in zip(valid_df['Id'], valid_df['Target'].str.split(' ')):\n    valid_dataset_info.append({\n        'path':os.path.join(path_to_train, name),\n        'labels':np.array([int(label) for label in labels])})\nvalid_dataset_info = np.array(valid_dataset_info)\nprint(train_dataset_info.shape, valid_dataset_info.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"10b81f14ad40c3701f0da6dc1cb6cf3b38f50363"},"cell_type":"markdown","source":"### Let's look on distribution of train and val parts:"},{"metadata":{"trusted":true,"_uuid":"a296b95022068de50c508f405f3904fcfd7a93c1"},"cell_type":"code","source":"fig, (ax1, ax2) = plt.subplots(1, 2, figsize = (10, 5))\ntrain_sum_vec = np.sum(np.stack(train_df['target_vec'].values, 0), 0)\nvalid_sum_vec = np.sum(np.stack(valid_df['target_vec'].values, 0), 0)\nax1.bar(n_keys, [train_sum_vec[k] for k in n_keys])\nax1.set_title('Training Distribution')\nax2.bar(n_keys, [valid_sum_vec[k] for k in n_keys])\n_ = ax2.set_title('Validation Distribution')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ddbc11bf72ecab4028380d64a08660a70a2da028"},"cell_type":"markdown","source":"### Create model\nOur model will have two Xception branches, each will return 2048 size vector and they will multiply before prediction:"},{"metadata":{"trusted":true,"_uuid":"64bba0326be38da7add3594002563bea1c9703db"},"cell_type":"code","source":"def create_model(input_shape, n_out):\n    inp_image = Input(shape=input_shape)\n    inp_mask = Input(shape=input_shape)\n    pretrain_model_image = InceptionResNetV2(include_top=False, weights='imagenet',pooling='max')\n    pretrain_model_image.name='xception_image'\n    pretrain_model_mask = InceptionResNetV2(include_top=False, weights='imagenet',pooling='max')\n    pretrain_model_mask.name='xception_mask'\n    \n    \n    x = Multiply()([pretrain_model_image(inp_image), pretrain_model_mask(inp_mask)])\n    out = Dense(n_out, activation='sigmoid')(x)\n    model = Model(inputs=[inp_image, inp_mask], outputs=[out])\n\n    return model","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"615c4c3ae5311478cd07f697533c030a6d21e077"},"cell_type":"markdown","source":"### F1 metric for progress monitoring from [this kernel](https://www.kaggle.com/guglielmocamporese/macro-f1-score-keras):"},{"metadata":{"trusted":true,"_uuid":"960881b72a39ec9911a1bd9f097c9bbfb439cd97"},"cell_type":"code","source":"import tensorflow as tf\ndef f1(y_true, y_pred):\n    y_pred = K.round(y_pred)\n    tp = K.sum(K.cast(y_true*y_pred, 'float'), axis=0)\n    tn = K.sum(K.cast((1-y_true)*(1-y_pred), 'float'), axis=0)\n    fp = K.sum(K.cast((1-y_true)*y_pred, 'float'), axis=0)\n    fn = K.sum(K.cast(y_true*(1-y_pred), 'float'), axis=0)\n\n    p = tp / (tp + fp + K.epsilon())\n    r = tp / (tp + fn + K.epsilon())\n\n    f1 = 2*p*r / (p+r+K.epsilon())\n    f1 = tf.where(tf.is_nan(f1), tf.zeros_like(f1), f1)\n    return K.mean(f1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dd32cc3ef1a767b4a959fc0aa3260aa402ee1846"},"cell_type":"markdown","source":"### Compile model\nCompile our model, note, we will use binary_crossentropy as loss (we have sigmoid output layer and multilabel task) ans two metrics: accuracy and f1:"},{"metadata":{"trusted":true,"_uuid":"e7296424196067a0e0a8f2383c99bd2bb76281c4"},"cell_type":"code","source":"keras.backend.clear_session()\n\nmodel = create_model(input_shape=(299,299,3), n_out=28)\nmodel.compile(loss='binary_crossentropy', optimizer='adam',metrics=['acc', f1])\n\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2fbffd71262d7d7e591ad5955feb6b0c5ee61737"},"cell_type":"markdown","source":"### Train model\nWe have a little bit big model (two Xception branches), 1 epoch trains around 1 hour, so we will train only 4 epochs (kaggle has 6 hours limit). Also, we will validate only on 10% of valid data, here it is not necessary in fact, but when you will try it by yourself, you will monitor quality based on val f1 score and val loss."},{"metadata":{"trusted":true,"_uuid":"0343374956494b98425448b7aa9092a037aee401"},"cell_type":"code","source":"epochs = 4; batch_size = 12\ncheckpointer = ModelCheckpoint(\n    '../working/Xception.model', \n    verbose=2, \n    save_best_only=True)\n\n\n# create train and valid datagens\ntrain_generator = train_datagen.create_train(\n    train_dataset_info, batch_size, (299,299,3))\nvalidation_generator = train_datagen.create_train(\n    valid_dataset_info, batch_size, (299,299,3))\nK.set_value(model.optimizer.lr, 0.0002)\n# train model\nhistory = model.fit_generator(\n    train_generator,\n    steps_per_epoch=len(train_df)//batch_size,\n    validation_data=validation_generator,\n    validation_steps=len(valid_df)//batch_size//10,\n    epochs=epochs, \n    verbose=1,\n    callbacks=[checkpointer])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"43471eb25b760c5f3934bf48a3a953240cc21336"},"cell_type":"markdown","source":"### Visualize history\nThere are a few epochs only, but in full variant it will be useful:"},{"metadata":{"trusted":true,"_uuid":"884be9cb5e6c81d0ba382bc436810831b37e3e3f"},"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(15,5))\nax[0].set_title('loss')\nax[0].plot(history.epoch, history.history[\"loss\"], label=\"Train loss\")\nax[0].plot(history.epoch, history.history[\"val_loss\"], label=\"Validation loss\")\nax[1].set_title('acc')\nax[1].plot(history.epoch, history.history[\"acc\"], label=\"Train acc\")\nax[1].plot(history.epoch, history.history[\"val_acc\"], label=\"Validation acc\")\nax[0].legend()\n_ = ax[1].legend()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2a52b306cd27fc030cfd3f7583009945de7828ac"},"cell_type":"markdown","source":"### Create submit"},{"metadata":{"trusted":true,"_uuid":"9d9ee4a8c2b1fc75b9f8d58855e33b115e92134d"},"cell_type":"code","source":"submit = pd.read_csv('../input/sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"782cdb0d614add0f41c11ea180bd45e5600d60fb"},"cell_type":"code","source":"%%time\npredicted = []\nfrom tqdm import tqdm_notebook\nfor name in tqdm(submit['Id']):\n    path = os.path.join('../input/test/', name)\n    image1, image2 = train_datagen.load_image(path, (299,299,3))\n    score_predict = model.predict([image1[np.newaxis], image2[np.newaxis]])[0]\n    label_predict = np.arange(28)[score_predict>=0.5]\n    str_predict_label = ' '.join(str(l) for l in label_predict)\n    predicted.append(str_predict_label)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"07043ed35ad009ce6edc017e4aa88ac82571a750"},"cell_type":"code","source":"submit['Predicted'] = predicted\nsubmit.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7351ee784f893c3cf9480a0117ea34364a32dbc8"},"cell_type":"markdown","source":"### Analysis of submission\nLet's look at our results, Which proteins occur most often in our submission?"},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":false,"trusted":true,"_uuid":"ebfadb6b6421806ad085232584f0c872bc19c6da"},"cell_type":"code","source":"name_label_dict = {\n    0:  \"Nucleoplasm\",  \n    1:  \"Nuclear membrane\",   \n    2:  \"Nucleoli\",   \n    3:  \"Nucleoli fibrillar center\",   \n    4:  \"Nuclear speckles\",\n    5:  \"Nuclear bodies\",   \n    6:  \"Endoplasmic reticulum\",   \n    7:  \"Golgi apparatus\",   \n    8:  \"Peroxisomes\",   \n    9:  \"Endosomes\",   \n    10:  \"Lysosomes\",   \n    11:  \"Intermediate filaments\",   \n    12:  \"Actin filaments\",   \n    13:  \"Focal adhesion sites\",   \n    14:  \"Microtubules\",   \n    15:  \"Microtubule ends\",   \n    16:  \"Cytokinetic bridge\",   \n    17:  \"Mitotic spindle\",   \n    18:  \"Microtubule organizing center\",   \n    19:  \"Centrosome\",   \n    20:  \"Lipid droplets\",   \n    21:  \"Plasma membrane\",   \n    22:  \"Cell junctions\",   \n    23:  \"Mitochondria\",   \n    24:  \"Aggresome\",   \n    25:  \"Cytosol\",   \n    26:  \"Cytoplasmic bodies\",   \n    27:  \"Rods & rings\"\n}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"68862b3d843757aa0c0bd0533be974da6eef6894"},"cell_type":"code","source":"submit['target_list'] = submit['Predicted'].map(lambda x: [int(a) for a in str(x).split(' ')])\nsubmit['target_vec'] = submit['target_list'].map(lambda ck: [i in ck for i in range(max_idx+1)])\nall_labels = list(chain.from_iterable(submit['target_list'].values))\nc_val = Counter(all_labels)\nn_keys = c_val.keys()\nmax_idx = max(n_keys)\nfor k,v in name_label_dict.items():\n    print(v, 'count:', c_val[k] if k in c_val else 0)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ce5940aab723ab2f352623a5be34585fa85efc89"},"cell_type":"markdown","source":"### Classes distribution:"},{"metadata":{"trusted":true,"_uuid":"3f666de2811a4d3ef701f5386ef2dea651bffa0b"},"cell_type":"code","source":"train_sum_vec = np.sum(np.stack(submit['target_vec'].values, 0), 0)\n_ = plt.bar(n_keys, [train_sum_vec[k] for k in n_keys])\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9eb2af9af05a21bb53c5b31ef88b82478231f282"},"cell_type":"markdown","source":"### Conclusion\nIt is only example how we can use data, but also it is not finish variant, we can improve several things:\n1. Train more! 4 epochs is not enough;\n2. Change Xception network to another one (InceptionV3, for example);\n3. You can add augmentation for your data;\n4. Play with merge layer (now it is Multiply), you can change it to Add or something else.\n\n#### Happy kaggling!"},{"metadata":{"trusted":true,"_uuid":"d5725780b59f002f55a7f82e0b78ab810cd052ef"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}