{"cells":[{"metadata":{},"cell_type":"markdown","source":"<center><img src=\"https://raw.githubusercontent.com/dimitreOliveira/MachineLearning/master/Kaggle/Understanding%20Clouds%20from%20Satellite%20Images/banner.png\" width=\"800\"></center>\n<h1><center>Understanding Clouds from Satellite Images</center></h1><p></p>\n<h2><center>Cloud Segmentation with utility scripts and Keras</center></h2>\n\n#### This kernel is to show the new feature on Kaggle, script notebooks. I'm not doing EDA here because I already did it on [my other kernel](https://www.kaggle.com/dimitreoliveira/understanding-clouds-eda-and-keras-u-net), the goal here is just to demonstrate how to use script notebook and how it can improve our work, making it faster and cleaner.\n#### I found this addition really cool as I always try to write clean and modular code, it always saves time later, this may be another push towards better software practices on data science projects.\n\nWhat you will find on the [script I made](https://www.kaggle.com/dimitreoliveira/cloud-images-segmentation-utillity-script):\n- All used dependencies\n- External repository codes (need internet option ON)\n- Seed function (to make model runs more reproducible)\n- Segmentation functions related to this competition\n- Multi-thread data process functions (to resize and apply transformations faster)\n- Model evaluation (training plots)\n- Model post-process (Set threshold and removing small masks)\n- Prediction evaluation (Generate metrics over predictions and sample evaluation)\n- Data generator\n- Learning rate schedulers\n\n##### If you have any request to update or add anything to the scripts please let me know in the comments.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Dependencies","execution_count":null},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":false,"_kg_hide-output":true,"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"# Link for the script https://www.kaggle.com/dimitreoliveira/cloud-images-segmentation-utillity-script\nfrom cloud_images_segmentation_utillity_script import *\nfrom keras.models import load_model\n\n!pip install tta-wrapper --quiet\n\nseed = 0\nseed_everything(seed)\nwarnings.filterwarnings(\"ignore\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Load data","execution_count":null},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_kg_hide-input":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/understanding_cloud_organization/train.csv')\nsubmission = pd.read_csv('../input/understanding_cloud_organization/sample_submission.csv')\n\n# Preprocecss data\ntrain['image'] = train['Image_Label'].apply(lambda x: x.split('_')[0])\ntrain['label'] = train['Image_Label'].apply(lambda x: x.split('_')[1])\nsubmission['image'] = submission['Image_Label'].apply(lambda x: x.split('_')[0])\ntest = pd.DataFrame(submission['image'].unique(), columns=['image'])\n\n# Create one column for each mask\ntrain_df = pd.pivot_table(train, index=['image'], values=['EncodedPixels'], columns=['label'], aggfunc=np.min).reset_index()\ntrain_df.columns = ['image', 'Fish_mask', 'Flower_mask', 'Gravel_mask', 'Sugar_mask']\n\nprint('Compete set samples:', len(train_df))\nprint('Test samples:', len(submission))\n\ndisplay(train.head())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Train and validation split","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train, X_val = train_test_split(train_df, test_size=0.2, random_state=seed)\nX_train['set'] = 'train'\nX_val['set'] = 'validation'\ntest['set'] = 'test'\n\nprint('Train samples: ', len(X_train))\nprint('Validation samples: ', len(X_val))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model parameters","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"BACKBONE = 'efficientnetb4'\nBATCH_SIZE = 4\nEPOCHS = 12\nLEARNING_RATE = 1e-5\nHEIGHT = 384\nWIDTH = 480\nCHANNELS = 3\nN_CLASSES = 4\nES_PATIENCE = 5\nRLROP_PATIENCE = 3\nDECAY_DROP = 0.5\nmodel_path = 'uNet_%s_%sx%s.h5' % (BACKBONE, HEIGHT, WIDTH)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"preprocessing = sm.get_preprocessing(BACKBONE)\n\naugmentation = albu.Compose([albu.HorizontalFlip(p=0.5),\n                             albu.VerticalFlip(p=0.5),\n                             albu.ShiftScaleRotate(rotate_limit=30, shift_limit=0.1, p=0.5)\n                            ])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Pre-process data","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train_base_path = '../input/understanding_cloud_organization/train_images/'\ntest_base_path = '../input/understanding_cloud_organization/test_images/'\ntrain_images_dest_path = 'base_dir/train_images/'\nvalidation_images_dest_path = 'base_dir/validation_images/'\ntest_images_dest_path = 'base_dir/test_images/'\n\n# Making sure directories don't exist\nif os.path.exists(train_images_dest_path):\n    shutil.rmtree(train_images_dest_path)\nif os.path.exists(validation_images_dest_path):\n    shutil.rmtree(validation_images_dest_path)\nif os.path.exists(test_images_dest_path):\n    shutil.rmtree(test_images_dest_path)\n    \n# Creating train, validation and test directories\nos.makedirs(train_images_dest_path)\nos.makedirs(validation_images_dest_path)\nos.makedirs(test_images_dest_path)\n\ndef preprocess_data(df, HEIGHT=HEIGHT, WIDTH=WIDTH):\n    '''\n    This function needs to be defined here, because it will be called with no arguments, \n    and must have the default parameters from the beggining of the notebook (HEIGHT and WIDTH)\n    '''\n    df = df.reset_index()\n    for i in range(df.shape[0]):\n        item = df.iloc[i]\n        image_id = item['image']\n        item_set = item['set']\n        if item_set == 'train':\n            preprocess_image(image_id, train_base_path, train_images_dest_path, HEIGHT, WIDTH)\n        if item_set == 'validation':\n            preprocess_image(image_id, train_base_path, validation_images_dest_path, HEIGHT, WIDTH)\n        if item_set == 'test':\n            preprocess_image(image_id, test_base_path, test_images_dest_path, HEIGHT, WIDTH)\n\n# Pre-procecss train set\npre_process_set(X_train, preprocess_data)\n\n# Pre-procecss validation set\npre_process_set(X_val, preprocess_data)\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Pre-procecss test set\npre_process_set(test, preprocess_data)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data generator","execution_count":null},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"train_generator = DataGenerator(\n                  directory=train_images_dest_path,\n                  dataframe=X_train,\n                  target_df=train,\n                  batch_size=BATCH_SIZE,\n                  target_size=(HEIGHT, WIDTH),\n                  n_channels=CHANNELS,\n                  n_classes=N_CLASSES,\n                  preprocessing=preprocessing,\n                  augmentation=augmentation,\n                  seed=seed)\n\nvalid_generator = DataGenerator(\n                  directory=validation_images_dest_path,\n                  dataframe=X_val,\n                  target_df=train,\n                  batch_size=BATCH_SIZE, \n                  target_size=(HEIGHT, WIDTH),\n                  n_channels=CHANNELS,\n                  n_classes=N_CLASSES,\n                  preprocessing=preprocessing,\n                  seed=seed)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model","execution_count":null},{"metadata":{"_kg_hide-input":false,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"model = sm.Unet(backbone_name=BACKBONE, \n                encoder_weights='imagenet',\n                classes=N_CLASSES,\n                activation='sigmoid',\n                input_shape=(None, None, CHANNELS))\n\ncheckpoint = ModelCheckpoint(model_path, monitor='val_loss', mode='min', save_best_only=True, save_weights_only=True)\nes = EarlyStopping(monitor='val_loss', mode='min', patience=ES_PATIENCE, restore_best_weights=True, verbose=1)\nrlrop = ReduceLROnPlateau(monitor='val_loss', mode='min', patience=RLROP_PATIENCE, factor=DECAY_DROP, min_lr=1e-7, verbose=1)\n\nmetric_list = [dice_coef, sm.metrics.iou_score]\ncallback_list = [checkpoint, es, rlrop]\noptimizer = RAdam(learning_rate=LEARNING_RATE, warmup_proportion=0.1)\n\n","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true,"collapsed":true},"cell_type":"code","source":"#STEP_SIZE_TRAIN = len(X_train)//BATCH_SIZE\n#STEP_SIZE_VALID = len(X_val)*2//BATCH_SIZE\n\n\nhistory = model.fit_generator(generator=train_generator,\n                              validation_data=valid_generator,\n                              callbacks=callback_list,\n                              epochs=20,\n                              workers=2,\n                              verbose=1).history","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.compile(optimizer=optimizer, loss=sm.losses.bce_dice_loss, metrics=metric_list)\nhistory = model.fit_generator(generator=train_generator,\n                              validation_data=valid_generator,\n                              callbacks=callback_list,\n                              epochs=30,\n                              workers=2,\n                              verbose=1).history","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Model loss graph","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plot_metrics(history, metric_list=['loss', 'dice_coef', 'iou_score'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Threshold and mask size tunning\n - Here we could use some kind of parameter search, but to simplify I'm using default values","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"class_names = ['Fish  ', 'Flower', 'Gravel', 'Sugar ']\nbest_tresholds = [.5, .5, .5, .35]\nbest_masks = [25000, 20000, 22500, 15000]\n\nfor index, name in enumerate(class_names):\n    print('%s treshold=%.2f mask size=%d' % (name, best_tresholds[index], best_masks[index]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model evaluation","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train_metrics = get_metrics(model, train, X_train, train_images_dest_path, best_tresholds, best_masks, seed=seed, preprocessing=preprocessing, set_name='Train')\ndisplay(train_metrics)\n\nvalidation_metrics = get_metrics(model, train, X_val, validation_images_dest_path, best_tresholds, best_masks, seed=seed, preprocessing=preprocessing, set_name='Validation')\ndisplay(validation_metrics)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Apply model to test set","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from tta_wrapper import tta_segmentation\n\nmodel = tta_segmentation(model, h_flip=True, v_flip=True, h_shift=(-10, 10), v_shift=(-10, 10), merge='mean')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"test_df = []\n\nfor i in range(0, test.shape[0], 300):\n    batch_idx = list(range(i, min(test.shape[0], i + 300)))\n    batch_set = test[batch_idx[0]: batch_idx[-1]+1]\n    \n    test_generator = DataGenerator(\n                      directory=test_images_dest_path,\n                      dataframe=batch_set,\n                      target_df=submission,\n                      batch_size=1, \n                      target_size=(HEIGHT, WIDTH),\n                      n_channels=CHANNELS,\n                      n_classes=N_CLASSES,\n                      preprocessing=preprocessing,\n                      seed=seed,\n                      mode='predict',\n                      shuffle=False)\n    \n    preds = model.predict_generator(test_generator)\n\n    for index, b in enumerate(batch_idx):\n        filename = test['image'].iloc[b]\n        image_df = submission[submission['image'] == filename].copy()\n        pred_masks = preds[index, ].round().astype(int)\n        pred_rles = build_rles(pred_masks, reshape=(350, 525))\n        image_df['EncodedPixels'] = pred_rles\n\n        ### Post procecssing\n        pred_masks_post = preds[index, ].astype('float32') \n        for class_index in range(N_CLASSES):\n            pred_mask = pred_masks_post[...,class_index]\n            pred_mask = post_process(pred_mask, threshold=best_tresholds[class_index], min_size=best_masks[class_index])\n            pred_masks_post[...,class_index] = pred_mask\n\n        pred_rles_post = build_rles(pred_masks_post, reshape=(350, 525))\n        image_df['EncodedPixels_post'] = pred_rles_post\n        ###\n        \n        test_df.append(image_df)\n\nsub_df = pd.concat(test_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Inspecting some of the validation set predictions\n\n## Without post-processing","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Choose 3 samples at random\nimages_to_inspect = np.random.choice(X_val['image'].unique(), 3, replace=False)\ninspect_set = train[train['image'].isin(images_to_inspect)].copy()\ninspect_set_temp = []\n\ninspect_generator = DataGenerator(\n                    directory=validation_images_dest_path,\n                    dataframe=inspect_set,\n                    target_df=train,\n                    batch_size=1, \n                    target_size=(HEIGHT, WIDTH),\n                    n_channels=CHANNELS,\n                    n_classes=N_CLASSES,\n                    preprocessing=preprocessing,\n                    seed=seed,\n                    mode='fit',\n                    shuffle=False)\n\npreds = model.predict_generator(inspect_generator)\n\nfor index, b in enumerate(range(len(preds))):\n    filename = inspect_set['image'].iloc[b]\n    image_df = inspect_set[inspect_set['image'] == filename].copy()\n    pred_masks = preds[index, ].round().astype(int)\n    pred_rles = build_rles(pred_masks, reshape=(350, 525))\n    image_df['EncodedPixels_pred'] = pred_rles\n    \n    ### Post procecssing\n    pred_masks_post = preds[index, ].astype('float32') \n    for class_index in range(N_CLASSES):\n        pred_mask = pred_masks_post[...,class_index]\n        pred_mask = post_process(pred_mask, threshold=best_tresholds[class_index], min_size=best_masks[class_index])\n        pred_masks_post[...,class_index] = pred_mask\n\n    pred_rles_post = build_rles(pred_masks_post, reshape=(350, 525))\n    image_df['EncodedPixels_pred_post'] = pred_rles_post\n    ###\n    inspect_set_temp.append(image_df)\n\n\ninspect_set = pd.concat(inspect_set_temp)\ninspect_predictions(inspect_set, images_to_inspect, validation_images_dest_path, pred_col='EncodedPixels_pred')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## With post-processing","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"inspect_predictions(inspect_set, images_to_inspect, validation_images_dest_path, pred_col='EncodedPixels_pred_post')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Inspecting some of the test set predictions\n\n## Without post-process","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Choose 5 samples at random\nimages_to_inspect_test =  np.random.choice(sub_df['image'].unique(), 4, replace=False)\ninspect_predictions(sub_df, images_to_inspect_test, test_images_dest_path)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## With post-process","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"inspect_predictions(sub_df, images_to_inspect_test, test_images_dest_path, label_col='EncodedPixels_post')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Regular submission","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"submission_df = sub_df[['Image_Label' ,'EncodedPixels']]\nsubmission_df.to_csv('submission.csv', index=False)\ndisplay(submission_df.head(10))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Submission with post processing","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"submission_df_post = sub_df[['Image_Label' ,'EncodedPixels_post']]\nsubmission_df_post.columns = ['Image_Label' ,'EncodedPixels']\nsubmission_df_post.to_csv('submission_post_1.csv', index=False)\ndisplay(submission_df_post.head(10))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Cleaning created directories\nif os.path.exists(train_images_dest_path):\n    shutil.rmtree(train_images_dest_path)\nif os.path.exists(validation_images_dest_path):\n    shutil.rmtree(validation_images_dest_path)\nif os.path.exists(test_images_dest_path):\n    shutil.rmtree(test_images_dest_path)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}