{"cells":[{"metadata":{"_kg_hide-input":false,"_kg_hide-output":false,"_uuid":"6243c7f9ee7da33a721c6c2db71d7d9c6c434038"},"cell_type":"markdown","source":"This is the first competition I've entered.  No illusions of placing top 3 out of the gate, but gaining some experience and learning a thing or two are high on the priority list.  Plus if some of my code can help, that's always a bonus.\n\nHere's what I've got so far..."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true,"scrolled":false},"cell_type":"code","source":"# Main imports, training data load, and label mapping\nimport pandas as pd\nimport numpy as np\nimport os\nimport pickle\nimport matplotlib.pyplot as plt\nimport tensorflow as tf\nimport keras.backend.tensorflow_backend as tfb\nimport seaborn as sbn\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import MultiLabelBinarizer, minmax_scale\n\nfrom keras import layers, regularizers, optimizers, callbacks\nfrom keras import backend as K\nfrom keras.utils import normalize\nfrom keras.utils.generic_utils import get_custom_objects\nfrom keras.preprocessing import image\nfrom keras_preprocessing.image import Iterator, ImageDataGenerator\nfrom keras.applications import ResNet50\nfrom keras.models import Model, load_model\nfrom keras.losses import binary_crossentropy\n\nfrom functools import partial\n\nbase_path = '../input/human-protein-atlas-image-classification/'\nmodel_path = '../input/actually-this-is-my-first-rodeo/'\n\n# Label map provided\nlabel_map = {0: 'Nucleoplasm',\n             1: 'Nuclear membrane',\n             2: 'Nucleoli',\n             3: 'Nucleoli fibrillar center',\n             4: 'Nuclear speckles',\n             5: 'Nuclear bodies',\n             6: 'Endoplasmic reticulum',\n             7: 'Golgi apparatus',\n             8: 'Peroxisomes',\n             9: 'Endosomes',\n             10: 'Lysosomes',\n             11: 'Intermediate filaments',\n             12: 'Actin filaments',\n             13: 'Focal adhesion sites',\n             14: 'Microtubules',\n             15: 'Microtubule ends',\n             16: 'Cytokinetic bridge',\n             17: 'Mitotic spindle',\n             18: 'Microtubule organizing center',\n             19: 'Centrosome',\n             20: 'Lipid droplets',\n             21: 'Plasma membrane',\n             22: 'Cell junctions',\n             23: 'Mitochondria',\n             24: 'Aggresome',\n             25: 'Cytosol',\n             26: 'Cytoplasmic bodies',\n             27: 'Rods & rings'}\n\nall_img_labels = pd.read_csv(\"../input/human-protein-atlas-image-classification/train.csv\")\nall_img_labels['Target'] = all_img_labels['Target'].apply(lambda x: set(map(int, x.split())))\n\nprint('Labels converted to sets of integers:\\n\\n', all_img_labels.head())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f8229426b7112291e74d55c2647c413682987a84"},"cell_type":"markdown","source":"Here's how I've been dealing with the multi-class labels:"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true,"scrolled":false},"cell_type":"code","source":"mlb = MultiLabelBinarizer()\nall_img_bin = pd.DataFrame(data=mlb.fit_transform(all_img_labels['Target']),\n                           index=all_img_labels['Id'],\n                           columns=label_map.values())\nprint(all_img_bin.head())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"91ec97cdfdcbd79cf20ef47128d99bc6d4d19cca"},"cell_type":"markdown","source":"Issues that I and others have noticed with this: the classes have been sorted lexicographically (i.e. 1, 10, 100, 2, ...) if you don't convert the incoming Target values to integers.  They won't line up exactly with our label map unless you do that.  Something to be aware of...\n\nThere's been a lot of good exploration of the data so far, and I don't want to rehash it.  Here's some other stuff I've been tinkering with:"},{"metadata":{"trusted":true,"_uuid":"1fd6506a6a46bb17fd0bac7577f16075bf90e344","scrolled":false},"cell_type":"code","source":"# Balancing class weights\nclass_counts = all_img_bin.sum(axis=0)\nclass_weights = {class_idx: (class_counts.sum() / (len(class_counts) * class_counts[class_idx]))\n                 for class_idx, class_name in enumerate(class_counts.index)}\n\nprint('Top 10 Label Weights\\n')\nprint(pd.Series(class_weights).sort_values(ascending=False)[:10])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"42dcf5189fdfe981399175cf2ce28051c042b06e"},"cell_type":"markdown","source":"Here's where I've seen a lot of questions regarding image/channel scaling or normalization.  The documentation I've seen regarding neural network inputs indicates that they should, as much as possible, be scaled to, \"small values.\"  The way I'm experimenting with it below involves scaling each channel that way per image, but I wonder if it needs to be normalized across all channels?  All images?\n\nAllunia's current exploration [here](https://www.kaggle.com/allunia/protein-atlas-exploration-and-baseline/notebook) finds that the color intensity across images is not consistent, which to me may indicate scaling each channel separately and each image separately may be a good idea.\n\nI've been able to keep as much of the original ImageDataGenerator intact as possible, so anyone can still pass to the IDG-Mod any keyword arguments that the original would take, which is an update to an earlier version of this class, which was boxed in to just one type of normalization hard-coded into the class.  I also added an argument for the path to the images it needs to load (e.g. 'train', 'test', etc.)."},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"7d008bef49c6c4c1e5b88148428a1b3eaa7bb57e","_kg_hide-input":true},"cell_type":"code","source":"# Messing with getting all channels into one array\nsample_id = all_img_bin.index.values[np.random.randint(len(all_img_bin))] # select random image ID from training set\n\ndef img_load(img_id, img_path, img_size):\n    img_array = np.zeros(shape=img_size)\n\n    for i, channel in enumerate(['_red.png', '_green.png', '_blue.png', '_yellow.png']):\n        sample_img_path = os.path.join(img_path, '{}{}'.format(img_id, channel))\n        sample_img_channel = image.load_img(sample_img_path, color_mode='grayscale',\n                                           target_size=img_size[:2])\n        img_array[:,:,i] = sample_img_channel\n\n    img_array = image.img_to_array(img_array)\n    img_array = normalize(img_array, axis=0)\n    \n    return img_array\n\nsample_img = img_load(img_id=sample_id, img_path=os.path.join(base_path, 'train'),\n                     img_size=(256, 256, 4))\ncolors = ['Reds', 'Greens', 'Blues', 'Oranges']\nfig, axes = plt.subplots(1, 4, figsize=(20, 20))\n\nprint('Sample image channels separated:')\n\nfor idx in range(len(colors)):\n    axes[idx].imshow(sample_img[:,:,idx], colors[idx])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6636495537ed002f2ef3daf7002dbcf03ee77960","scrolled":false,"_kg_hide-input":true},"cell_type":"code","source":"# Subclassing the keras ImageDataGenerator to fit our needs\n\nclass ImageDataGenMod(ImageDataGenerator):\n    \n    def __init__(self, **kwargs):\n        super().__init__(**kwargs)\n        \n    def flow_from_gen(self, x_set, data_path, y_set=None,\n                     batch_size=32, target_size=(512, 512), color_mode='rgba',\n                     data_format='channels_last', **kwargs):\n        return SimpleImageGen(x_set, self, data_path, y_set, target_size, color_mode,\n                              data_format, batch_size, **kwargs)\n\n\nclass SimpleImageGen(Iterator):\n    \n    def __init__(self, x_set, image_data_generator, data_path, y_set=None,\n                 target_size=(512, 512), color_mode='rgba',\n                 data_format='channels_last', batch_size=32,\n                 shuffle=True, seed=None,\n                 save_to_dir=None, save_prefix='', save_format='png',\n                 subset=None, interpolation='nearest'):\n        super().common_init(image_data_generator,\n                            target_size,\n                            color_mode,\n                            data_format,\n                            save_to_dir,\n                            save_prefix,\n                            save_format,\n                            subset,\n                            interpolation)\n        self.x, self.y = x_set, y_set\n        self.batch_size = batch_size\n        self.target_size = target_size\n        self.color_mode = color_mode\n        self.samples = len(self.x)\n        self.colors = ['_red.png', '_green.png', '_blue.png', '_yellow.png']\n        self.data_path = data_path\n        \n        super().__init__(self.samples,\n                        batch_size,\n                        shuffle,\n                        seed)\n    \n    def __len__(self):\n        return int(np.ceil(len(self.x) / float(self.batch_size)))\n    \n    def _get_batches_of_transformed_samples(self, index_array):\n        #print(index_array)\n        batch_x = np.zeros((len(index_array),) + self.image_shape)\n        if self.y is not None:\n            batch_y = self.y[index_array]\n            #print(batch_y)\n        \n        for n, idx in enumerate(index_array):\n            img_id = self.x[idx]\n            for i in range(len(self.color_mode)):\n                img_path = os.path.join(self.data_path, img_id + self.colors[i])\n                img_channel = image.load_img(img_path, color_mode='grayscale',\n                                            target_size=self.target_size)\n                batch_x[n, :, :, i] = img_channel\n            \n            batch_x[n] = image.img_to_array(batch_x[n])\n            \n            params = self.image_data_generator.get_random_transform(batch_x[n].shape)\n            batch_x[n] = self.image_data_generator.apply_transform(batch_x[n], params)\n            batch_x[n] = self.image_data_generator.standardize(batch_x[n])\n        \n        if self.y is not None:\n            return batch_x, np.array(batch_y)\n        else:\n            return batch_x\n        \n    def next(self):\n        with self.lock:\n            index_array = next(self.index_generator)\n        \n        return self._get_batches_of_transformed_samples(index_array)\n\n\n# Flow test...\ndata_gen_args = dict(preprocessing_function=partial(normalize, axis=0),\n                     rotation_range=45,\n                     width_shift_range=0.2,\n                     height_shift_range=0.2,\n                     shear_range=0.2,\n                     zoom_range=0.2,\n                     channel_shift_range=0.2,\n                     brightness_range=(0.3, 1.0),\n                     horizontal_flip=True,\n                     vertical_flip=True,\n                     fill_mode='constant',\n                     cval=0)\n\ntest_gen = ImageDataGenMod(**data_gen_args)\ntest_data = test_gen.flow_from_gen(x_set=all_img_bin.index.values,\n                                   data_path=os.path.join(base_path, 'train'),\n                                   y_set=all_img_bin.values,\n                                   batch_size=5)\n\ntest_batch = next(test_data)\nprint('Modded ImageDataGenerator test:\\n')\nprint('Data batch shape: ' + str(test_batch[0].shape),\n      'Labels batch shape: ' + str(test_batch[1].shape),\n      sep='\\n')\n\nfig, axes = plt.subplots(1, 5, figsize=(20, 20))\n\nfor idx in range(len(test_batch[0])):\n    axes[idx].imshow(test_batch[0][idx])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8e31e1c0ae29131546d1d8f58975220cf8823176"},"cell_type":"markdown","source":"Noice.\n\nI suppose we could get a base model going to see what we get.  Now that I've been able to successfully subclass the IDG (as it seems to work so far), it already has image augmentation built in to help out.  You don't have to augment your images...but it helps.\n\nI've seen this metric in a bunch of places, but I believe the original credit belongs to [Guglielmo Camporese](https://www.kaggle.com/guglielmocamporese/macro-f1-score-keras):"},{"metadata":{"trusted":true,"_uuid":"3b07185aa35e9b1dcd84d0c4120bf10833309a01"},"cell_type":"code","source":"# Macro F1 loss/metrics functions\ndef f1_metric(y_true, y_pred):\n    y_pred = K.round(y_pred)\n\n    tp = K.sum(K.cast(y_true * y_pred, 'float'), axis=0)\n    fp = K.sum(K.cast((1 - y_true) * y_pred, 'float'), axis=0)\n    fn = K.sum(K.cast(y_true * (1 - y_pred), 'float'), axis=0)\n    \n    p = tp / (tp + fp + K.epsilon())\n    r = tp / (tp + fn + K.epsilon())\n    \n    f1 = 2 * p * r / (p + r + K.epsilon())\n    f1 = tf.where(tf.is_nan(f1), tf.zeros_like(f1), f1)\n    return K.mean(f1)\n\ndef f1_loss(y_true, y_pred):\n    \n    tp = K.sum(K.cast(y_true * y_pred, 'float'), axis=0)\n    fp = K.sum(K.cast((1 - y_true) * y_pred, 'float'), axis=0)\n    fn = K.sum(K.cast(y_true * (1 - y_pred), 'float'), axis=0)\n    \n    p = tp / (tp + fp + K.epsilon())\n    r = tp / (tp + fn + K.epsilon())\n    \n    f1 = 2 * p * r / (p + r + K.epsilon())\n    f1 = tf.where(tf.is_nan(f1), tf.zeros_like(f1), f1)\n    return 1 - K.mean(f1)\n\n# Found this custom weighted binary crossentropy loss function here:\n# https://stackoverflow.com/questions/42158866/neural-network-for-multi-label-classification-with-large-number-of-classes-outpu/47313183#47313183\nPOS_WEIGHT = 10 # Seems arbitrary, needs to be tuned\n\ndef weighted_binary_crossentropy(target, output):\n    # transform back to logits\n    _epsilon = tfb._to_tensor(tfb.epsilon(), output.dtype.base_dtype)\n    output = tf.clip_by_value(output, _epsilon, 1 - _epsilon)\n    output = tf.log(output / (1 - output))\n    \n    # compute weighted loss\n    loss = tf.nn.weighted_cross_entropy_with_logits(targets=target,\n                                                    logits=output,\n                                                    pos_weight=POS_WEIGHT)\n    return tf.reduce_mean(loss, axis=-1)\n\ndef combo_loss(y_true, y_pred):\n    f1_loss_comp = f1_loss(y_true, y_pred)\n    bce_loss_comp = binary_crossentropy(y_true, y_pred)\n    return f1_loss_comp + bce_loss_comp","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3faecf30514a36ba788ed8658a7bc701b634042c"},"cell_type":"markdown","source":"Time for a model...  I'm fiddling with both a hand-built version and some pre-trained models such as ResNet50.  "},{"metadata":{"_kg_hide-input":true,"trusted":true,"scrolled":false,"_uuid":"606aaece24925e8435a3980ba6d256353ca92c9a"},"cell_type":"code","source":"# If I've already saved a model, load it, otherwise build a new one\ntest_shape = (256, 256, 4)\nregs = regularizers.l1_l2(l1=0.001, l2=0.001)\nopt = optimizers.Adam(lr=1e-6)\n\nbuild_new = False\n\nif not build_new:\n    get_custom_objects().update({'weighted_binary_crossentropy': weighted_binary_crossentropy,\n                                 'combo_loss': combo_loss,\n                                 'f1_loss': f1_loss,\n                                 'f1_metric': f1_metric})\n    model = load_model(os.path.join(model_path, 'hpa_model.h5'))\n    print('Model loaded...')\nelse:\n    model_in = layers.Input(test_shape)\n\n    # Basic model build...    \n    ### 2x2 Features\n    model_in2 = layers.SeparableConv2D(32, (2, 2), padding='same', kernel_regularizer=regs)(model_in)\n    model_in2 = layers.BatchNormalization()(model_in2)\n    model_in2 = layers.Activation('relu')(model_in2)\n    model_in2 = layers.SeparableConv2D(32, (2, 2), padding='same', kernel_regularizer=regs)(model_in2)\n    model_in2 = layers.BatchNormalization()(model_in2)\n    model_in2 = layers.Activation('relu')(model_in2)\n    model_in2 = layers.MaxPool2D((2, 2), name='maxpool_2')(model_in2)\n    \n    model_in2 = layers.SeparableConv2D(64, (2, 2), padding='same', kernel_regularizer=regs)(model_in2)\n    model_in2 = layers.BatchNormalization()(model_in2)\n    model_in2 = layers.Activation('relu')(model_in2)\n    model_in2 = layers.MaxPool2D((2, 2))(model_in2)\n    \n    model_in2 = layers.SeparableConv2D(128, (2, 2), padding='same', kernel_regularizer=regs)(model_in2)\n    model_in2 = layers.BatchNormalization()(model_in2)\n    model_in2 = layers.Activation('relu')(model_in2)\n    model_in2 = layers.MaxPool2D((2, 2))(model_in2)\n    \n    model_in2 = layers.SeparableConv2D(256, (2, 2), padding='same', kernel_regularizer=regs)(model_in2)\n    model_in2 = layers.BatchNormalization()(model_in2)\n    model_in2 = layers.Activation('relu')(model_in2)\n    model_in2 = layers.MaxPool2D((2, 2))(model_in2)\n    \n    ### 4x4 Features\n    model_in4 = layers.SeparableConv2D(32, (4, 4), padding='same', kernel_regularizer=regs)(model_in)\n    model_in4 = layers.BatchNormalization()(model_in4)\n    model_in4 = layers.Activation('relu')(model_in4)\n    model_in4 = layers.SeparableConv2D(32, (4, 4), padding='same', kernel_regularizer=regs)(model_in4)\n    model_in4 = layers.BatchNormalization()(model_in4)\n    model_in4 = layers.Activation('relu')(model_in4)\n    model_in4 = layers.MaxPool2D((2, 2), name='maxpool_4')(model_in4)\n    \n    model_in4 = layers.SeparableConv2D(64, (4, 4), padding='same', kernel_regularizer=regs)(model_in4)\n    model_in4 = layers.BatchNormalization()(model_in4)\n    model_in4 = layers.Activation('relu')(model_in4)\n    model_in4 = layers.MaxPool2D((2, 2))(model_in4)\n    \n    model_in4 = layers.SeparableConv2D(128, (4, 4), padding='same', kernel_regularizer=regs)(model_in4)\n    model_in4 = layers.BatchNormalization()(model_in4)\n    model_in4 = layers.Activation('relu')(model_in4)\n    model_in4 = layers.MaxPool2D((2, 2))(model_in4)\n    \n    model_in4 = layers.SeparableConv2D(256, (4, 4), padding='same', kernel_regularizer=regs)(model_in4)\n    model_in4 = layers.BatchNormalization()(model_in4)\n    model_in4 = layers.Activation('relu')(model_in4)\n    model_in4 = layers.MaxPool2D((2, 2))(model_in4)\n    \n    ### 8x8 Features\n    model_in8 = layers.SeparableConv2D(32, (8, 8), padding='same', kernel_regularizer=regs)(model_in)\n    model_in8 = layers.BatchNormalization()(model_in8)\n    model_in8 = layers.Activation('relu')(model_in8)\n    model_in8 = layers.SeparableConv2D(32, (8, 8), padding='same', kernel_regularizer=regs)(model_in8)\n    model_in8 = layers.BatchNormalization()(model_in8)\n    model_in8 = layers.Activation('relu')(model_in8)\n    model_in8 = layers.MaxPool2D((2, 2), name='maxpool_8')(model_in8)\n    \n    model_in8 = layers.SeparableConv2D(64, (8, 8), padding='same', kernel_regularizer=regs)(model_in8)\n    model_in8 = layers.BatchNormalization()(model_in8)\n    model_in8 = layers.Activation('relu')(model_in8)\n    model_in8 = layers.MaxPool2D((2, 2))(model_in8)\n    \n    model_in8 = layers.SeparableConv2D(128, (8, 8), padding='same', kernel_regularizer=regs)(model_in8)\n    model_in8 = layers.BatchNormalization()(model_in8)\n    model_in8 = layers.Activation('relu')(model_in8)\n    model_in8 = layers.MaxPool2D((2, 2))(model_in8)\n    \n    model_in8 = layers.SeparableConv2D(256, (8, 8), padding='same', kernel_regularizer=regs)(model_in8)\n    model_in8 = layers.BatchNormalization()(model_in8)\n    model_in8 = layers.Activation('relu')(model_in8)\n    model_in8 = layers.MaxPool2D((2, 2))(model_in8)\n    \n    ### 16x16 Features\n    model_in16 = layers.SeparableConv2D(32, (16, 16), padding='same', kernel_regularizer=regs)(model_in)\n    model_in16 = layers.BatchNormalization()(model_in16)\n    model_in16 = layers.Activation('relu')(model_in16)\n    model_in16 = layers.SeparableConv2D(32, (16, 16), padding='same', kernel_regularizer=regs)(model_in16)\n    model_in16 = layers.BatchNormalization()(model_in16)\n    model_in16 = layers.Activation('relu')(model_in16)\n    model_in16 = layers.MaxPool2D((2, 2), name='maxpool_16')(model_in16)\n    \n    model_in16 = layers.SeparableConv2D(64, (16, 16), padding='same', kernel_regularizer=regs)(model_in16)\n    model_in16 = layers.BatchNormalization()(model_in16)\n    model_in16 = layers.Activation('relu')(model_in16)\n    model_in16 = layers.MaxPool2D((2, 2))(model_in16)\n    \n    model_in16 = layers.SeparableConv2D(128, (16, 16), padding='same', kernel_regularizer=regs)(model_in16)\n    model_in16 = layers.BatchNormalization()(model_in16)\n    model_in16 = layers.Activation('relu')(model_in16)\n    model_in16 = layers.MaxPool2D((2, 2))(model_in16)\n    \n    model_in16 = layers.SeparableConv2D(256, (16, 16), padding='same', kernel_regularizer=regs)(model_in16)\n    model_in16 = layers.BatchNormalization()(model_in16)\n    model_in16 = layers.Activation('relu')(model_in16)\n    model_in16 = layers.MaxPool2D((2, 2))(model_in16)\n\n    # Concatenate\n    model_body = layers.Concatenate()([model_in2,\n                                      model_in4,\n                                      model_in8,\n                                      model_in16])\n    \n    model_body = layers.SeparableConv2D(1024, (3, 3), padding='same', kernel_regularizer=regs)(model_body)\n    model_body = layers.BatchNormalization()(model_body)\n    model_body = layers.Activation('relu')(model_body)\n    model_body = layers.MaxPool2D((2, 2))(model_body)\n    \n    model_body = layers.Flatten()(model_body)\n\n    model_out = layers.Dropout(0.5)(model_body)\n    model_out = layers.Dense(256, kernel_regularizer=regs)(model_out)\n    model_out = layers.BatchNormalization()(model_out)\n    model_out = layers.Activation('relu')(model_out)\n    model_out = layers.Dense(28, activation='sigmoid')(model_out)\n\n    model = Model(model_in, model_out)\n\n    model.compile(loss=combo_loss,\n                 optimizer=opt,\n                 metrics=[f1_metric, 'acc'])\n    model.save('hpa_model.h5')\n    print('Model built...')\n\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ebdfb5c4386114e11bce3fc54e90fd16306e6d42","scrolled":true},"cell_type":"code","source":"# Split data into training and validation sets\ntrain_val_split = 0.15\nbatch_size = 16\n\nx_train, x_test, y_train, y_test = train_test_split(all_img_bin.index.values,\n                                                   all_img_bin.values,\n                                                   test_size=train_val_split,\n                                                   random_state=0)\n\nprint('Training set shape: ', x_train.shape, y_train.shape)\nprint('Validation set shape: ', x_test.shape, y_test.shape)\n\ntrain_datagen = ImageDataGenMod(**data_gen_args)\ntest_datagen = ImageDataGenMod(preprocessing_function=partial(normalize, axis=0))\n\ntrain_gen = train_datagen.flow_from_gen(x_set=x_train,\n                                        data_path=os.path.join(base_path, 'train'),\n                                        y_set=y_train,\n                                        target_size=test_shape[:2],\n                                        batch_size=batch_size,\n                                       color_mode='rgba')\nval_gen = test_datagen.flow_from_gen(x_set=x_test,\n                                     data_path=os.path.join(base_path, 'train'),\n                                     y_set=y_test,\n                                     target_size=test_shape[:2],\n                                     batch_size=batch_size,\n                                    color_mode='rgba')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6f1e5fb56ac165bdd35484231bcc9df01d8e7e2c"},"cell_type":"markdown","source":"So far my models take a really long time to train, but they seem to improve over many epochs and have the ability to differentiate between multiple labels, but do pretty poorly against the test set."},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"1b6ccbca2b5171302ac2c038b33b5024fe74054a","_kg_hide-output":false,"_kg_hide-input":true},"cell_type":"code","source":"# Train the model on augmented images, plot the loss and F1 over epochs\ncbacks = [callbacks.ReduceLROnPlateau(monitor='val_loss',\n                                      mode='min',\n                                      factor=0.5,\n                                      patience=1,\n                                      verbose=1,\n                                      cooldown=2,\n                                      min_lr=1e-10),\n         callbacks.ModelCheckpoint(filepath='hpa_model.h5',\n                                   monitor='val_f1_metric',\n                                   mode='max',\n                                   save_best_only=True,\n                                   verbose=1)]\n\nif not build_new:\n    with open(os.path.join(model_path, 'all_history.pickle'), 'rb') as pkl:\n        all_history = pickle.load(pkl)\n    print('History loaded')\nelse:\n    all_history = {'metric': [],\n                  'loss': [],\n                  'val_metric': [],\n                  'val_loss': []}\n    print('History created')\n\nhistory = model.fit_generator(generator=train_gen,\n                              steps_per_epoch=len(train_gen),\n                              epochs=4,\n                              validation_data=val_gen,\n                              validation_steps=len(val_gen),\n                              class_weight=class_weights,\n                              callbacks=cbacks)\n\nall_history['metric'].extend(history.history['f1_metric'])\nall_history['loss'].extend(history.history['loss'])\nall_history['val_metric'].extend(history.history['val_f1_metric'])\nall_history['val_loss'].extend(history.history['val_loss'])\n\nwith open('all_history.pickle', 'wb') as pkl:\n    pickle.dump(all_history, pkl)\n\n# Plot the model fitting\nstart_index = 40\n\nmetric = all_history['metric'][-start_index:]\nval_metric = all_history['val_metric'][-start_index:]\nloss = all_history['loss'][-start_index:]\nval_loss = all_history['val_loss'][-start_index:]\nepochs = range(1, len(metric) + 1)\n\nfig, axes = plt.subplots(1, 2)\naxes[0].plot(epochs, metric, 'bo', label='Training F1')\naxes[0].plot(epochs, val_metric, 'b', label='Validation F1')\naxes[0].legend()\naxes[1].plot(epochs, loss, 'bo', label='Training Loss')\naxes[1].plot(epochs, val_loss, 'b', label='Validation Loss')\naxes[1].legend()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3b8b89b31992037e0792cc66a4d64289a5a962b4"},"cell_type":"markdown","source":"A sample of what the model predicts for the test set:"},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"d4dbd8a42ac3ca297ffd7c80737b44b8a7d3ad18","_kg_hide-input":false},"cell_type":"code","source":"# Make predictions with the model and output to a submission file\ntest_ids = pd.read_csv(os.path.join(base_path, 'sample_submission.csv'))['Id']\n\ntest_preds = test_datagen.flow_from_gen(x_set=list(test_ids),\n                                        data_path=os.path.join(base_path, 'test'),\n                                        target_size=test_shape[:2],\n                                        batch_size=batch_size,\n                                        color_mode='rgba')\npreds = model.predict_generator(test_preds,\n                               steps=len(test_preds))\ntarget = [' '.join(map(str, x)) for x in mlb.inverse_transform(np.round(preds))]\n\noutput = pd.DataFrame(data=list(zip(test_ids, target)),\n                      columns=['Id','Predicted'])\nprint(output.head())\n\noutput.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1c3b6fa9cdc9dc37c9757915626c559a98e4027a"},"cell_type":"markdown","source":"One great thing about convolutional neural networks is that they're much less of a black box than perhaps a multi-layer perceptron network, i.e. it's pretty easy to get a visual into what the model is doing.  Here's the top 3 pooling layers from this particular model looking at a sample test image.  Each box is one of the filters from that layer.  Based on how each filter is being activated, you can start to see what parts of the image will play a role in its predicted label(s).\n\nIf you haven't picked up Francois Chollet's book on how to do this stuff (\"Deep Learning with Python\"), I highly recommend it."},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"943832d3dd0a79c73a5bf8ad4b60e3e912979dd1","_kg_hide-input":true},"cell_type":"code","source":"# What's the model 'seeing?'\n# Pick random test image ID\nview_id = test_ids[np.random.randint(len(test_ids))]\nview_img = img_load(img_id=view_id, img_path=os.path.join(base_path, 'test'),\n                   img_size=test_shape)\n\nlayer_depth = 4\nimages_per_row = 8\n\nview_img_tensor = np.expand_dims(view_img, axis=0)\n\nplt.imshow(view_img_tensor[0])\nplt.title('Test Image (RGBA)')\nplt.show()\n\n# Get the outputs of the initial pooling layers\nlayer_outputs = [layer.output for layer in model.layers if 'maxpool' in layer.name][:layer_depth]\nactivation_model = Model(inputs=model.input, outputs=layer_outputs)\nactivations = activation_model.predict(view_img_tensor)\n\n# Looking at some layers and channels of the model\nlayer_names = [layer.name for layer in model.layers if 'maxpool' in layer.name][:layer_depth]\n\nfor layer_name, layer_activation in zip(layer_names, activations):\n    n_features = layer_activation.shape[-1]\n    \n    size = layer_activation.shape[1]\n    \n    n_cols = n_features // images_per_row\n    display_grid = np.zeros((size * n_cols, size * images_per_row))\n    \n    for col in range(n_cols):\n        for row in range(images_per_row):\n            channel_img = layer_activation[0, :, :, col * images_per_row + row]\n            \n            display_grid[col * size : (col + 1) * size,\n                        row * size : (row + 1) * size] = channel_img\n    \n    scale = 2.0 / size\n    plt.figure(figsize=(scale * display_grid.shape[1],\n                       scale * display_grid.shape[0]))\n    plt.title(layer_name)\n    plt.grid(False)\n    plt.imshow(display_grid, aspect='auto', cmap='viridis')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"501316fa2d5c3ea113bd4582739d1845e8e1c687"},"cell_type":"markdown","source":"Next I want to see if the labels the model is predicting are at least reasonable.  By that I would hope the distribution of predicted labels would be similar to the training labels distribution (assuming the test data is reasonably similar to the training set).  Also, the number of labels per image should be within the limits present in the training set, which in this case is somewhere between 1 and 4 labels assigned to each image."},{"metadata":{"trusted":true,"_uuid":"bd24e955d491501632429cf15868af695896ab1a","_kg_hide-input":true},"cell_type":"code","source":"# Are the predictions reasonable to the training data?\nfig, axes = plt.subplots(1, 2, figsize=(20, 7.5), sharey=True)\n\nsbn.barplot(data=np.round(preds), ax=axes[0]).set_title('Label Distribution (Predicted)')\nsbn.barplot(data=all_img_bin.values, ax=axes[1]).set_title('Label Distribution (Training)')\n\nplt.show()\n\nnum_pred, counts_pred = np.unique(np.sum(np.round(preds), axis=1), return_counts=True)\nnum_train, counts_train = np.unique(np.sum(all_img_bin.values, axis=1), return_counts=True)\n\nfig, axes = plt.subplots(1, 2, figsize=(20, 7.5), sharey=True)\n\nsbn.barplot(x=num_pred, y=counts_pred, ax=axes[0]).set_title('Labels/Image (Predicted)')\nsbn.barplot(x=num_train, y=counts_train, ax=axes[1]).set_title('Labels/Image (Training)')\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"57fd0486be35799d3baaf652d5c41619d7a7d7fc"},"cell_type":"markdown","source":"Clearly these models aren't gonna cut it..."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}