{"cells":[{"metadata":{"_uuid":"baa36a17c6f16a41bff863e7a27f33588f6b9f38"},"cell_type":"markdown","source":"**Disclaimer**: This is a **WIP**. \n\nSome of the things that I am still working on:\n\n* Cleaning up the code and finishing the TODOs. => some progress.\n* Making MLflow work (the package should be added to the docker image).\n* Adding more details about setting up a remote MLflow server (AWS or another instance setup + launching the server).\n* More documentation and explanations. => some progress.\n* Extracting reusable sections and putting them into the Kaggle tools library.\n* Some refactoring and better variable names.\n* The loss is negative. Check if this is expected or not.\n* (and probably other things...)\n\n**There is something wrong with the hyperparamters returned at the end (some values are missing). I wil fix this as soon as possible. I am making this notebook public nonetheless. \n**\nIn the meantime, feel free to leave comments and/or suggestions. Thanks for you time. Enjoy!"},{"metadata":{"_uuid":"1ac883829c31d7878c22cf531c13658f28aa672e"},"cell_type":"markdown","source":"# Overview\n\n\nIn this notebook, I use the MLflow library to manage the various hyperopt runs (each run is defined by some hyperparameters drawn from a space).\n\nNotice that the processing and modelling parts are based on this [notebook](https://www.kaggle.com/hmendonca/u-net-model-with-submission), which is it self a fork of this [one](https://www.kaggle.com/kmader/baseline-u-net-model-part-1). \n\nIn order to run this notebook, you will need to install mlflow (it seems that this isn't enough). In the meantime, I am sending a PR to make this available in the [docker image](https://github.com/Kaggle/docker-python).\n\nFinally, I will (later) add a wrapper for MLflow and hyperopt in the [kaggle-tools](https://github.com/yassineAlouini/kaggle-tools) library. \n\nLet's go!"},{"metadata":{"_uuid":"7e0a8b327e211dd8b7e165bd0f579c5fdcf7d63e"},"cell_type":"markdown","source":"# Libraries imports\n\nAs usual some libraries imports for processing, modelling, and hyperparameters' optimization. There is obviously the [MLflow](https://github.com/mlflow/mlflow) library as well. "},{"metadata":{"trusted":true,"_uuid":"35049cd78223505740a527ed6447e8bf38e9d7f8"},"cell_type":"code","source":"# Data loading and processing imports\nimport os\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nfrom skimage.io import imread\nfrom skimage.segmentation import mark_boundaries\nfrom skimage.util.montage import montage2d as montage\nfrom skimage.morphology import binary_opening, disk\nfrom skimage.morphology import label\nimport gc\nimport datetime\n\n# Hp optimization imports\nfrom hyperopt import STATUS_OK, Trials, fmin, hp, tpe\n\n\n# Modelling imports\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras import models, layers\nimport keras.backend as K\nfrom keras.optimizers import Adam\nfrom keras.losses import binary_crossentropy\n# Some useful Keras callbacks\nfrom keras.callbacks import ModelCheckpoint, LearningRateScheduler, EarlyStopping, ReduceLROnPlateau\nfrom sklearn.model_selection import train_test_split\n\n# TODO: Make this work\n# !pip install mlflow and using the Packages installer in the kernel aren't enough. :(","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0a4ce78c5ab5f8bf173d62aab87abe8ddc25adb8","collapsed":true},"cell_type":"code","source":"# Enable garbage collection\ngc.enable()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9dd0dbfd3fd1ef14c4ae0f7a8accbb209c454802","collapsed":true},"cell_type":"code","source":"# Global constants\nSEED = 31415\n# Set it to a small number so that it can run in this notebook.\n# Notice that if it is too small, hyperopt behaves as random selection.\nMAX_EVALS = 1\nCUSTOM_DICE_LOSS_EPSILON = 1e-3\n# TODO: Use pathlib later\nBASE_DATA_PATH = \"../input/\"\nMASKS_DATA_PATH = os.path.join(BASE_DATA_PATH, 'train_ship_segmentations.csv')\nTRAIN_IMAGES_FOLDER = os.path.join(BASE_DATA_PATH, 'train')\nTEST_IMAGES_FOLDER = os.path.join(BASE_DATA_PATH, 'test')\n# According to the data description, some files from the test folder shoud be ignore\nTEST_IMGS_TO_IGNORE = ['13703f040.jpg',\n '14715c06d.jpg',\n '33e0ff2d5.jpg',\n '4d4e09f2a.jpg',\n '877691df8.jpg',\n '8b909bb20.jpg',\n 'a8d99130e.jpg',\n 'ad55c3143.jpg',\n 'c8260c541.jpg',\n 'd6c7f17c7.jpg',\n 'dc3e7c901.jpg',\n 'e44dffe88.jpg',\n 'ef87bad36.jpg',\n 'f083256d8.jpg']\n# These two patiences thresholds are small so that this notebook can run with limited resources\nREDUCE_LR_PATIENCE = 2\nEARLY_STOPPING_PATIENCE = 2\n# Fraction of the validation size (compared to the total train size)\nVALID_SIZE = 0.3\n# Minimum size (in KB) of files to keep\nFILE_SIZE_KB_THRESHOLD = 50\n# The original size of the image\n# TODO: Check if it is really 3 channels.\nIMG_SIZE = (768, 768, 3)\n# downsampling in preprocessing\n# TODO: Should these be hp to optimize as well?\nIMG_SCALING = (4, 4)\nEDGE_CROP = 16\n# downsampling inside the network\nNET_SCALING = (1, 1)\n# number of validation images to use\nVALID_IMG_COUNT = 600\n# maximum number of steps_per_epoch in training\nMAX_TRAIN_STEPS = 150\nMAX_TRAIN_EPOCHS = 10\n# The hyperparameters space over which to search. \n# TODO: Improve the ranges and the used distributions to sample.\nHYPERPARAMETERS_SPACE = {\n        # TODO: What is the best scale for Gaussian noise?\n        'gaussian_noise': hp.choice('gaussian_noise', [0.1, 0.2, 0.3]),\n        'batch_size':  hp.choice('batch_size', [8, 16, 32, 64, 128]),\n        'upsample_mode': hp.choice('upsmaple_mode', [\"SIMPLE\", \"DECONV\"]),\n        'augment_brightness': hp.choice('augment_brightness', [True, False]),\n        'max_train_steps': MAX_TRAIN_STEPS, \n        'max_train_epochs': MAX_TRAIN_EPOCHS, \n        'valid_img_count': VALID_IMG_COUNT,\n        'img_scaling': IMG_SCALING,\n        'edge_crop': EDGE_CROP,\n        'net_scaling': NET_SCALING\n    }\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"136bac866110bdd22f9f78b232dc9522dd18a164"},"cell_type":"markdown","source":"# Preprocessing and postprocessing utils "},{"metadata":{"_uuid":"b4480984f8571cc1d91e330aa2ac6321ed1fc048"},"cell_type":"markdown","source":"In order to keep the code organized, I have grouped the image processing functions in the next cell. "},{"metadata":{"trusted":true,"_uuid":"bc8c56650d7d9fed79f54fc5e5d74c0e22096331","collapsed":true},"cell_type":"code","source":"# The classic RLE encoding code, from the great: https://www.kaggle.com/paulorzp/run-length-encode-and-decode\ndef rle_encode(img, min_threshold=1e-3, max_threshold=None):\n    '''\n    img: numpy array, 1 - mask, 0 - background\n    Returns run length as string formated\n    '''\n    if np.max(img) < min_threshold:\n        return '' ## no need to encode if it's all zeros\n    if max_threshold and np.mean(img) > max_threshold:\n        return '' ## ignore overfilled mask\n    pixels = img.T.flatten()\n    pixels = np.concatenate([[0], pixels, [0]])\n    runs = np.where(pixels[1:] != pixels[:-1])[0] + 1\n    runs[1::2] -= runs[::2]\n    return ' '.join(str(x) for x in runs)\n\ndef rle_decode(mask_rle, shape=(768, 768)):\n    '''\n    mask_rle: run-length as string formated (start length)\n    shape: (height,width) of array to return \n    Returns numpy array, 1 - mask, 0 - background\n    '''\n    s = mask_rle.split()\n    starts, lengths = [np.asarray(x, dtype=int) for x in (s[0:][::2], s[1:][::2])]\n    starts -= 1\n    ends = starts + lengths\n    img = np.zeros(shape[0]*shape[1], dtype=np.uint8)\n    for lo, hi in zip(starts, ends):\n        img[lo:hi] = 1\n    return img.reshape(shape).T  # Needed to align to RLE direction\n\ndef masks_as_image(in_mask_list):\n    # Take the individual ship masks and create a single mask array for all ships\n    # TODO: This mask size shouldn't be hardcoded (768, 768).\n    all_masks = np.zeros((768, 768), dtype = np.uint8)\n    for mask in in_mask_list:\n        if isinstance(mask, str):\n            all_masks |= rle_decode(mask)\n    return all_masks\n\ndef masks_as_color(in_mask_list):\n    # Take the individual ship masks and create a color mask array for each ships\n    # TODO: This mask size shouldn't be hardcoded (768, 768).\n    all_masks = np.zeros((768, 768), dtype = np.float)\n    scale = lambda x: (len(in_mask_list)+x+1) / (len(in_mask_list)*2) ## scale the heatmap image to shift \n    for i,mask in enumerate(in_mask_list):\n        if isinstance(mask, str):\n            all_masks[:,:] += scale(i) * rle_decode(mask)\n    return all_masks\n\n# TODO: Add some documentation\ndef make_image_gen(input_df, batch_size, img_scaling):\n    df = input_df.copy()\n    all_batches = list(df.groupby('ImageId'))\n    out_rgb = []\n    out_mask = []\n    while True:\n        np.random.shuffle(all_batches)\n        for c_img_id, c_masks in all_batches:\n            rgb_path = os.path.join(TRAIN_IMAGES_FOLDER, c_img_id)\n            c_img = imread(rgb_path)\n            c_mask = masks_as_image(c_masks['EncodedPixels'].values)\n            c_mask = np.expand_dims(c_mask, axis=-1)\n            if img_scaling is not None:\n                c_img = c_img[::img_scaling[0], ::img_scaling[1]]\n                c_mask = c_mask[::img_scaling[0], ::img_scaling[1]]\n            out_rgb += [c_img]\n            out_mask += [c_mask]\n            if len(out_rgb)>=batch_size:\n                yield np.stack(out_rgb, 0)/255.0, np.stack(out_mask, 0)\n                out_rgb, out_mask=[], []\n                \n# TODO: Add some documentation for the augmentation pipeline as well.\n# TODO: Finish this and add some documentation.\ndef build_image_generator(augment_brightness):\n    \"\"\" Build an image data generator (for images and labels). \n    For more details about this class, check the documentation here: \n    https://keras.io/preprocessing/image/.\n    \"\"\"\n    # TODO: Describe what each data augementation parameter does.\n    data_generator_dict = dict(featurewise_center = False, \n                               samplewise_center = False,\n                               rotation_range = 45, \n                               width_shift_range = 0.1, \n                               height_shift_range = 0.1, \n                               shear_range = 0.01,\n                               zoom_range = [0.9, 1.25],  \n                               horizontal_flip = True, \n                               vertical_flip = True,\n                               fill_mode = 'reflect',\n                               data_format = 'channels_last')\n    # brightness can be problematic since it seems to change the labels differently from the images \n    if augment_brightness:\n        data_generator_dict['brightness_range'] = [0.5, 1.5]\n    image_gen = ImageDataGenerator(**data_generator_dict)\n\n    if augment_brightness:\n        data_generator_dict.pop('brightness_range')\n    label_gen = ImageDataGenerator(**data_generator_dict)\n    return image_gen, label_gen\n\n# TODO: Add some documentation and improve variables names.\ndef create_aug_gen(in_gen, augment_brightness, seed = None):\n    image_gen, label_gen = build_image_generator(augment_brightness)\n    np.random.seed(seed if seed is not None else np.random.choice(range(9999)))\n    for in_x, in_y in in_gen:\n        seed = np.random.choice(range(9999))\n        # keep the seeds syncronized otherwise the augmentation to the images is different from the masks\n        g_x = image_gen.flow(255*in_x, \n                             batch_size = in_x.shape[0], \n                             seed = seed, \n                             shuffle=True)\n        g_y = label_gen.flow(in_y, \n                             batch_size = in_x.shape[0], \n                             seed = seed, \n                             shuffle=True)\n\n        yield next(g_x)/255.0, next(g_y)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"914829c2cd7f7277ab86b8fc853a3fdc28f9de70"},"cell_type":"markdown","source":"# MLflow setup"},{"metadata":{"_uuid":"fc28b603f7caf10e15a50e0ca82cb4443e793565"},"cell_type":"markdown","source":"I will be using an EC2 instance for tracking and storing models.\nFor more details on how to setup an instance, check this [databricks notebook](https://docs.databricks.com/spark/latest/mllib/mlflow.html)."},{"metadata":{"trusted":true,"_uuid":"4811be0b994777e4f171309ef50d9acc1e563aaf"},"cell_type":"code","source":"URI = \"your/remote/instance\"\nEXPERIMENT_NAME = \"airbus_ship_detection_u_net\"\ntry:\n    import mlflow\n    mlflow.set_tracking_uri(URI)\n    EXPERIMENT_ID = mlflow.create_experiment(EXPERIMENT_NAME)\nexcept (ImportError, ModuleNotFoundError):\n    print(\"Unfortunately, MLflow isn't available. :(\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"991185adc30236eb45218647d4511c358f1cecde"},"cell_type":"markdown","source":"# Hyperparameters optimization setup"},{"metadata":{"_uuid":"8ef35c88c82ee79f8662e6f15b00e35f1e486ab0"},"cell_type":"markdown","source":"In order to run hyperopt, one needs two things:\n\n1. A function that returns a loss to minimize\n2. A function that optimizes this loss\n\nThe first function, `get_loss`,  takes hyperparameters in, builds a segmentation model (a [Unet](https://arxiv.org/abs/1505.04597) variation), and trains this model on a validation set then it output a loss. \n\nThe second function, `optimize`, suggests hyperarameters to the `get_loss` (selected from the `HYPERPARAMETERS_SPACE` possibilites), and finds the optimal ones (those that return the lowest loss). \n\nI have grouped these into a class, `HyperoptHPOptimizer`, so that I can use it later once the model pipeline is ready. "},{"metadata":{"trusted":true,"_uuid":"3351278f44afa941a14a3905192095b0dc5b786a","collapsed":true},"cell_type":"code","source":"class HyperoptHPOptimizer(object):\n    \n    def __init__(self, generate_model_history, hyperparameters_space, max_evals):\n        # TODO: Add some documentation\n        self.generate_model_history = generate_model_history\n        self.trials = Trials()\n        self.max_evals = max_evals\n        self.hyperparameters_space = hyperparameters_space\n\n    def _get_loss_with_mlflow(self, hyperparameters):\n        # MLflow will track and save hyperparameters, loss, and scores. \n        with mlflow.run(experiment_id=EXPERIMENT_ID):\n            print(\"Training with the following hyperparameters: \")\n            print(hyperparameters)\n            for k, v in hyperparameters.iteritems():\n                mlflow.log_param(k, v)\n            history = self.generate_model_history(hyperparameters)\n            # Log the various losses and metrics (on train and validation)\n            for k, v in history.history.items():\n                mlflow.log_metric(k, v[-1])\n            # Use the last validation loss from the history object to optimize\n            loss = history.history[\"val_loss\"][-1]\n            return {'loss': loss, 'status': STATUS_OK}\n        \n    def _get_loss_without_mlflow(self, hyperparameters):\n            print(\"Training with the following hyperparameters: \")\n            print(hyperparameters)\n            history = self.generate_model_history(hyperparameters)\n            # Use the last validation loss from the history object to optimize\n            loss = history.history[\"val_loss\"][-1]\n            return {'loss': loss, 'status': STATUS_OK}\n    \n    def get_loss(self, hyperparameters):\n        try:\n            import mlflow\n            return self._get_loss_with_mlflow(hyperparameters)\n        except (ImportError, ModuleNotFoundError):\n            return self._get_loss_without_mlflow(hyperparameters)\n\n    def optimize(self):\n        \"\"\"\n        This is the optimization function that given a space of \n        hyperparameters and a scoring function, finds the best hyperparameters.\n        \"\"\"\n        # Use the fmin function from Hyperopt to find the best hyperparameters\n        # Here we use the tree-parzen estimator method. \n        best = fmin(self.get_loss, self.hyperparameters_space, algo=tpe.suggest, \n                    trials=self.trials,  max_evals=self.max_evals)\n        return best","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"60b6790c8411c3563b6ce6dbf49f5ae7869c7be6"},"cell_type":"markdown","source":"# Load data and rebalance it"},{"metadata":{"trusted":true,"_uuid":"39d80a72302985cf11995b12a7d183ba0f54d197","scrolled":true,"collapsed":true},"cell_type":"code","source":"# TODO: Use the https://www.kaggle.com/yassinealouini/idiomatic-pandas-processing?scriptVersionId=5308185\n# to cleanup this part. \n\n\ndef get_data(file_size_kb_threshold = FILE_SIZE_KB_THRESHOLD):\n    \"\"\" Load and process train and validation data (images and masks).\n    \"\"\"\n    # Two vectorized functions\n    _v_path_join = np.vectorize(os.path.join)\n    _v_file_size = np.vectorize(lambda fp: (os.stat(fp).st_size) / 1024)\n\n    # Read the masks DataFrame\n    masks_df = pd.read_csv(MASKS_DATA_PATH)\n    print(masks_df.shape)\n    ships_df = (masks_df.groupby('ImageId')[\"EncodedPixels\"]\n                        .count()\n                        .reset_index()\n                        .rename(columns={\"EncodedPixels\": \"ships\"})\n                        .assign(has_ship=lambda df: np.where(df['ships'] > 0, 1, 0))\n                        .assign(file_path=lambda df: _v_path_join(TRAIN_IMAGES_FOLDER, \n                                                                  df.ImageId.astype(str)))\n                        .assign(file_size_kb=lambda df: _v_file_size(df.file_path))\n                        .loc[lambda df: df.file_size_kb > file_size_kb_threshold, :])\n\n    print(ships_df.head())\n    train_ids, valid_ids = train_test_split(ships_df, \n                     test_size = VALID_SIZE, \n                     stratify = ships_df['ships'])\n    train_df = pd.merge(masks_df, train_ids)\n    valid_df = pd.merge(masks_df, valid_ids)\n\n\n    print(train_df.shape[0], 'training masks')\n    print(valid_df.shape[0], 'validation masks')\n    \n    # Rebalance the training DataFrame.\n    # TODO: Improve the rebalancing code.\n    train_df['grouped_ship_count'] = train_df['ships'].map(lambda x: (x+2)//3)\n    balanced_train_df = train_df.groupby('grouped_ship_count').apply(lambda x: x.sample(1500))\n    return balanced_train_df, valid_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"33300c4f03b6600da7b418f775d11d7ebf76a35a"},"cell_type":"code","source":"# Garbage collection time!\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ba08494eb9736ec3556b7c879143cdcdea89febf"},"cell_type":"markdown","source":"# Build a Model\n\nThis is a variation of the U-Net model"},{"metadata":{"trusted":true,"_uuid":"2687377309d3cbbab1197f4eccd2b50ab996f5a6","collapsed":true},"cell_type":"code","source":"# Build U-Net model\n\n# TODO: Document the various used hyperparameters\ndef build_u_net_model(input_shape, upsample_mode=\"DECONV\", gaussian_noise=0.1, \n                      padding=\"same\", net_scaling=None, img_scaling=IMG_SCALING, *args, **kargs):\n    # TODO: Move these to the utils section?\n    def _upsample_conv(filters, kernel_size, strides, padding):\n        return layers.Conv2DTranspose(filters, kernel_size, strides=strides, \n                                      padding=padding)\n    def _upsample_simple(filters, kernel_size, strides, padding):\n        return layers.UpSampling2D(strides)    \n    \n    upsample_dict = {\"DECONV\": _upsample_conv, \"SIMPLE\": _upsample_simple}\n    \n    upsample = upsample_dict.get(upsample_mode, _upsample_simple)\n\n    input_img = layers.Input(input_shape, name = 'RGB_Input')\n    pp_in_layer = input_img\n    \n    # TODO: Add dropout for regularization?\n\n    # Some preprocessing\n    # TODO: Add explanation of the different stes.\n    if net_scaling is not None:\n        pp_in_layer = layers.AvgPool2D(net_scaling)(pp_in_layer)\n\n    pp_in_layer = layers.GaussianNoise(gaussian_noise)(pp_in_layer)\n    pp_in_layer = layers.BatchNormalization()(pp_in_layer)\n\n    c1 = layers.Conv2D(8, (3, 3), activation='relu', padding=padding) (pp_in_layer)\n    c1 = layers.Conv2D(8, (3, 3), activation='relu', padding=padding) (c1)\n    p1 = layers.MaxPooling2D((2, 2)) (c1)\n\n    c2 = layers.Conv2D(16, (3, 3), activation='relu', padding=padding) (p1)\n    c2 = layers.Conv2D(16, (3, 3), activation='relu', padding=padding) (c2)\n    p2 = layers.MaxPooling2D((2, 2)) (c2)\n\n    c3 = layers.Conv2D(32, (3, 3), activation='relu', padding=padding) (p2)\n    c3 = layers.Conv2D(32, (3, 3), activation='relu', padding=padding) (c3)\n    p3 = layers.MaxPooling2D((2, 2)) (c3)\n\n    c4 = layers.Conv2D(64, (3, 3), activation='relu', padding=padding) (p3)\n    c4 = layers.Conv2D(64, (3, 3), activation='relu', padding=padding) (c4)\n    p4 = layers.MaxPooling2D(pool_size=(2, 2)) (c4)\n\n\n    c5 = layers.Conv2D(128, (3, 3), activation='relu', padding=padding) (p4)\n    c5 = layers.Conv2D(128, (3, 3), activation='relu', padding=padding) (c5)\n\n    u6 = upsample(64, (2, 2), strides=(2, 2), padding=padding) (c5)\n    u6 = layers.concatenate([u6, c4])\n    c6 = layers.Conv2D(64, (3, 3), activation='relu', padding=padding) (u6)\n    c6 = layers.Conv2D(64, (3, 3), activation='relu', padding=padding) (c6)\n\n    u7 = upsample(32, (2, 2), strides=(2, 2), padding=padding) (c6)\n    u7 = layers.concatenate([u7, c3])\n    c7 = layers.Conv2D(32, (3, 3), activation='relu', padding=padding) (u7)\n    c7 = layers.Conv2D(32, (3, 3), activation='relu', padding=padding) (c7)\n\n    u8 = upsample(16, (2, 2), strides=(2, 2), padding=padding) (c7)\n    u8 = layers.concatenate([u8, c2])\n    c8 = layers.Conv2D(16, (3, 3), activation='relu', padding=padding) (u8)\n    c8 = layers.Conv2D(16, (3, 3), activation='relu', padding=padding) (c8)\n\n    u9 = upsample(8, (2, 2), strides=(2, 2), padding=padding) (c8)\n    u9 = layers.concatenate([u9, c1], axis=3)\n    c9 = layers.Conv2D(8, (3, 3), activation='relu', padding=padding) (u9)\n    c9 = layers.Conv2D(8, (3, 3), activation='relu', padding=padding) (c9)\n\n    d = layers.Conv2D(1, (1, 1), activation='sigmoid') (c9)\n    # TODO: Why is this commented\n    # d = layers.Cropping2D((EDGE_CROP, EDGE_CROP))(d)\n    # d = layers.ZeroPadding2D((EDGE_CROP, EDGE_CROP))(d)\n    if net_scaling is not None:\n        d = layers.UpSampling2D(net_scaling)(d)\n\n    seg_model = models.Model(inputs=[input_img], outputs=[d])\n    if img_scaling is not None:\n        fullres_model = models.Sequential()\n        fullres_model.add(layers.AvgPool2D(img_scaling, \n                                           input_shape = (None, None, 3)))\n        fullres_model.add(seg_model)\n        fullres_model.add(layers.UpSampling2D(img_scaling))\n    else:\n        fullres_model = seg_model\n    fullres_model.summary()\n    return fullres_model","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d5fb41e9d7c72d07042f5d450a4b5681da5b6d26"},"cell_type":"markdown","source":"# Define metric and loss"},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"56b010a5ae154410bd7e7b92fc496111869c3827"},"cell_type":"code","source":"def dice_metric(y_true, y_pred, smooth=1):\n    \"\"\"\n    Also known as the Sorensen-Dice coeffecient (https://en.wikipedia.org/wiki/S%C3%B8rensen%E2%80%93Dice_coefficient),\n    this is the F1 score (i.e. harmonic mean of precision and recall). \n    Notice that this metric has a smoothness parameter (smooth) to avoid division by 0.\n    \"\"\"\n    intersection = K.sum(y_true * y_pred, axis=[1,2,3])\n    union = K.sum(y_true, axis=[1,2,3]) + K.sum(y_pred, axis=[1,2,3])\n    # Compute the dice metric and then take the average over the samples.\n    return K.mean((2. * intersection + smooth) / (union + smooth), axis=0)\n\n\ndef custom_dice_loss(in_gt, in_pred):\n    \"\"\" This is a custom loss function that has two contributions: binary crossentropy \n    (this is the usual metric used for binary classification) and - the dice metric (to turn it into a loss).\n    \"\"\"\n    return CUSTOM_DICE_LOSS_EPSILON * binary_crossentropy(in_gt, in_pred) - dice_metric(in_gt, in_pred)\n\ndef true_positive_rate_metric(y_true, y_pred):\n    \"\"\" TPR (true positive rate) measures the ratio of true positives over positives.\n    Notice the round step so that the predicted values are transformed into 0 or 1 values instead of floats\n    in the range [0, 1]. \n    \"\"\"\n    return K.sum(K.flatten(y_true) * K.flatten(K.round(y_pred))) / K.sum(y_true)\n\n\n\ndef f2_metric():\n    # TODO: Add the F2 metric. \n    pass\n\nMETRICS = [true_positive_rate_metric, \n           dice_metric, \n           \"binary_accuracy\"]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1678069aa8013510264ba898291c6ae2dce88a76","collapsed":true},"cell_type":"code","source":"def get_compiled_model(hyperparameters, input_shape=IMG_SIZE):\n    model = build_u_net_model(input_shape, **hyperparameters)\n    # TODO: This should be in the hp list as well.\n    learning_rate = 1e-3\n    adam_optimizer = Adam(learning_rate, decay=1e-6)\n    model.compile(optimizer=adam_optimizer, loss=custom_dice_loss, metrics=METRICS)\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7282d18de3aff1cee12ff89b7d511a391702814f","collapsed":true},"cell_type":"code","source":"# TODO: Add some documentation for these callbacks.\n\nweight_path = \"best_weights.h5\"\n\n# TODO: Move some of the hyperparameters to the constants list\ncheckpoint = ModelCheckpoint(weight_path, monitor='val_loss', \n                             verbose=1, save_best_only=True, \n                             mode='min', save_weights_only=True)\n\nreduceLROnPlat = ReduceLROnPlateau(monitor='val_loss', factor=0.2,\n                                   patience=REDUCE_LR_PATIENCE, verbose=1, mode='min',\n                                   min_delta=0.0001, cooldown=2, min_lr=1e-7)\n# probably needs to be more patient, but kaggle time is limited\nearly = EarlyStopping(monitor=\"val_loss\", mode=\"min\", verbose=2,\n                      patience=EARLY_STOPPING_PATIENCE)\n\nCALLBACKS = [checkpoint, early, reduceLROnPlat]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5b67d808c0b8c7e28bff41e6d3858ff6f09dd626","scrolled":false,"collapsed":true},"cell_type":"code","source":"# TODO: Add some documentation\ndef _generate_model_history(input_train_df, input_valid_df, hyperparameters, n_samples, input_shape):\n    # Copy input DataFrames to avoid side-effects\n    train_df = input_train_df.copy()\n    valid_df = input_valid_df.copy()\n    max_train_steps = hyperparameters[\"max_train_steps\"]\n    batch_size = hyperparameters[\"batch_size\"]\n    img_scaling = hyperparameters[\"img_scaling\"]\n    max_train_epochs = hyperparameters[\"max_train_epochs\"]\n    valid_img_count =  hyperparameters[\"valid_img_count\"]\n    augment_brightness = hyperparameters[\"augment_brightness\"]\n    steps_per_epoch = min(max_train_steps, n_samples//batch_size)\n    print(\"Using {} steps per epoch.\".format(steps_per_epoch))\n    img_genarator = make_image_gen(train_df, batch_size, img_scaling)\n    augmented_img_generator = create_aug_gen(img_genarator, augment_brightness)\n    # TODO: Improve the names of these returned values.\n    valid_x, valid_y = next(make_image_gen(valid_df, valid_img_count, img_scaling))\n    model = get_compiled_model(hyperparameters, input_shape)\n    # Use only one worker for thread-safety reason. \n    # TODO: Investigate this claim.\n    return model.fit_generator(augmented_img_generator, steps_per_epoch=steps_per_epoch,\n                                epochs=max_train_epochs, validation_data=(valid_x, valid_y),\n                                callbacks=CALLBACKS, workers=1)\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7003f38eec39697b43a2e3c1fe7d68a06cd80f7c"},"cell_type":"markdown","source":"# Putting everything together"},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"55d5ed5d4a298a07a90ad3b4af0554e5a2a426c3","collapsed":true},"cell_type":"code","source":"train_df, valid_df = get_data()\nn_samples = train_df.shape[0]\ninput_shape = IMG_SIZE","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"07d0b04ac0c021cba1ff8d592c862e7deb091cdf"},"cell_type":"code","source":"# We wrap the _generate_model_history so that the train_df DataFrame isn't reconstructed for each iteration.\ngenerate_model_history = lambda hp: _generate_model_history(train_df, valid_df, hp, n_samples, input_shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"76056352c3b0133649a85d62a7942a51fea32af9","collapsed":true},"cell_type":"code","source":"hp_optimizer = HyperoptHPOptimizer(generate_model_history, hyperparameters_space=HYPERPARAMETERS_SPACE, \n                                   max_evals=MAX_EVALS)\noptimal_hyperparameters = hp_optimizer.optimize()\nprint(optimal_hyperparameters)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"96b80f6d73455dd126868d789393ebbc62a51cb6"},"cell_type":"markdown","source":"# Save the optimal model\n\nOnce the best hyperparameters have been found, build a model pipeline with these hyperparameters and save it. "},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"6daac235d04fda7b81357678365510f5dc52c060"},"cell_type":"code","source":"def save_optimal_model(input_train_df, input_valid_df, hyperparameters, n_samples, input_shape, \n                       model_path):\n    # Copy input DataFrames to avoid side-effects\n    train_df = input_train_df.copy()\n    valid_df = input_valid_df.copy()\n    max_train_steps = hyperparameters[\"max_train_steps\"]\n    batch_size = hyperparameters[\"batch_size\"]\n    img_scaling = hyperparameters[\"img_scaling\"]\n    max_train_epochs = hyperparameters[\"max_train_epochs\"]\n    valid_img_count =  hyperparameters[\"valid_img_count\"]\n    augment_brightness = hyperparameters[\"augment_brightness\"]\n    steps_per_epoch = min(max_train_steps, n_samples//batch_size)\n    print(\"Using {} steps per epoch.\".format(steps_per_epoch))\n    img_genarator = make_image_gen(train_df, batch_size, img_scaling)\n    augmented_img_generator = create_aug_gen(img_genarator, augment_brightness)\n    # TODO: Improve the names of these returned values.\n    valid_x, valid_y = next(make_image_gen(valid_df, valid_img_count, img_scaling))\n    model = get_compiled_model(hyperparameters, input_shape)\n    model.fit_generator(augmented_img_generator, steps_per_epoch=steps_per_epoch,\n                        epochs=max_train_epochs, validation_data=(valid_x, valid_y),\n                        callbacks=CALLBACKS, workers=1)\n    model.save(model_path)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"434106486438ccb884ce9090f632ad6e5fb95909","collapsed":true},"cell_type":"code","source":"# It is usually a good idea to add the creation datetime in the model name\ncurrent_datetime = datetime.datetime.now()\nmodel_path = \"optimal_custom_u_net_model.h5\".format(current_datetime.strftime(\"%Y-%m-%d-%H-%M\"))\nsave_optimal_model(train_df, valid_df, optimal_hyperparameters, n_samples, input_shape, model_path)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"565637bca752c78e9c7302f91a273d265f7f05e3"},"cell_type":"markdown","source":"Now that the model is saved, you can reload it and use it to make a submission. Good luck!"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}