{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Kuzushiji recognition with MaskRCNN using Keras/Tensorflow\n#### This kernel is here to establish the pipeline to use a MaskRCNN on the Kuzushiji competition dataset, even though there is little chance it can obtain great results in the processing time allowed on kernels. It is using the [Matterplot implementation](https://github.com/matterport/Mask_RCNN) of MaskRCNN and is inspired from [this kernel](https://www.kaggle.com/pednoi/training-mask-r-cnn-to-be-a-fashionista-lb-0-07) from the iMaterialist competition. One of the key steps that is introduced in this kernel is the data preparation which consists of generating masks using an Otsu threshold.\n### If you find this kernel useful, please give an upvote! :)"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport gc\nimport sys\nimport time\nimport json\nimport glob\nimport random\nfrom pathlib import Path\nimport pandas as pd\n\nfrom PIL import Image\nimport cv2\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nfrom imgaug import augmenters as iaa\n\nimport itertools\nfrom tqdm import tqdm","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Filter the dataset to make the task easier to learn"},{"metadata":{},"cell_type":"markdown","source":"Instead of loading the .csv file initially given, we have already generated the masks for all files for the 57 most encountered symbols. Therefore, the preprocessing steps are mainly presented to provide a way to process the rest of the symbols' masks if needed. Tests with the most frequent 649 symbols did not proved to work at all, mainly because it requires a crazy amount of time to train."},{"metadata":{"trusted":true},"cell_type":"code","source":"training_df = pd.read_csv(\"../input/rle-kuzushiji-dataset-57-most-freq-symbols/RLE_dataset_57most_freq_symbols.csv\")\n# training_df = pd.read_csv(\"../input/rle-kuzushiji-dataset-649-most-freq-symbols/dataset_with_rle_649classes.csv\")\nunicode_map = {codepoint: char for codepoint, char in pd.read_csv('../input/kuzushiji-recognition/unicode_translation.csv').values}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"frequency_dict = {}\nboundaries_dict = {}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Not the nicest looking piece of code but it basically gets the number of samples per kuzushiji symbol. (Also getting the size of the samples too but not using it further)"},{"metadata":{"trusted":true},"cell_type":"code","source":"for idx, row in training_df.iterrows():\n    try:\n        page_labels = np.array(row[\"labels\"].split(' ')).reshape(-1, 5)\n    except:\n        pass\n    for symbol_info in page_labels:\n        symbol_unicode = symbol_info[0]\n        x, y, boundary_width, boundary_height  = [int(item) for item in symbol_info[1:]]\n        if not frequency_dict.get(symbol_unicode):\n            frequency_dict[symbol_unicode] = 1\n            boundaries_dict[symbol_unicode] = [boundary_width*boundary_height]\n        else:\n            frequency_dict[symbol_unicode] += 1\n            boundaries_dict[symbol_unicode].append(boundary_width*boundary_height)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### A quick plot reveals a highly imbalanced dataset."},{"metadata":{},"cell_type":"markdown","source":"When looking at the plot, we quickly realise that a lot of kuzushiji have few samples. We may want to get rid of some of them to start."},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.plot(frequency_dict.values())\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We set a threshold to only keep the classes with a high number of examples. Of course, by doing this, we assume the distribution in the training and test data are similar.\nPS: The threshold is very high as an attempt to improve learning within the time allowed on kernels."},{"metadata":{"trusted":true},"cell_type":"code","source":"count=0\ntotal_samples=0\nthreshold=2000\nfor kuzushiji_count in frequency_dict.values():\n    if kuzushiji_count>threshold:\n        count+=1\n        total_samples+=kuzushiji_count\nprint(\"Number of classes kept:\",count)\nprint(\"Percentage of the examples these classes represent:\",100*total_samples/sum(frequency_dict.values()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"kuzushiji_to_detect = []\nfor unicode, kuzushiji_count in frequency_dict.items():\n    if kuzushiji_count>threshold:\n        kuzushiji_to_detect.append(unicode)\n        \n#Added the sort as the model would not seem to learn and only learn background masks instead of the actual classes' masks.\nkuzushiji_to_detect = sorted(kuzushiji_to_detect)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Setting up the environment for the MaskRCNN"},{"metadata":{"trusted":true},"cell_type":"code","source":"DATA_DIR = Path('../kaggle/input')\nROOT_DIR = \"../../working\"\n\nNUM_CATS = len(kuzushiji_to_detect)\nIMAGE_SIZE = 512","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!git clone https://www.github.com/matterport/Mask_RCNN.git\nos.chdir('Mask_RCNN')\n\n!rm -rf .git # to prevent an error when the kernel is committed\n!rm -rf images assets # to prevent displaying images at the bottom of a kernel","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sys.path.append(ROOT_DIR+'/Mask_RCNN')\nfrom mrcnn.config import Config\n\nfrom mrcnn import utils\nimport mrcnn.model as modellib\nfrom mrcnn import visualize\nfrom mrcnn.model import log","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!wget --quiet https://github.com/matterport/Mask_RCNN/releases/download/v2.0/mask_rcnn_coco.h5\n!ls -lh mask_rcnn_coco.h5\n\nCOCO_WEIGHTS_PATH = 'mask_rcnn_coco.h5'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class KuzushijiConfig(Config):\n    NAME = \"kuzushiji\"\n    NUM_CLASSES = NUM_CATS + 1 # +1 for the background class\n    \n    GPU_COUNT = 1\n    IMAGES_PER_GPU = 4 # a memory error occurs when IMAGES_PER_GPU is too high\n    \n    BACKBONE = 'resnet50'\n    \n    IMAGE_MIN_DIM = IMAGE_SIZE\n    IMAGE_MAX_DIM = IMAGE_SIZE    \n    IMAGE_RESIZE_MODE = 'none'\n    \n    RPN_ANCHOR_SCALES = (16, 32, 64, 128, 256)\n    \n    # STEPS_PER_EPOCH should be the number of instances \n    # divided by (GPU_COUNT*IMAGES_PER_GPU), and so should VALIDATION_STEPS;\n    # however, due to the time limit, I set them so that this kernel can be run in 9 hours\n    STEPS_PER_EPOCH = 360\n    VALIDATION_STEPS = 40\n    \nconfig = KuzushijiConfig()\nconfig.display()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Create the dataset"},{"metadata":{},"cell_type":"markdown","source":"Just taking a subsample as it takes quite a long time to create the masks and I want to process the whole pipeline on this kernel."},{"metadata":{"trusted":true},"cell_type":"code","source":"category_dict = {}\nunique_symbols = kuzushiji_to_detect\nfor category_id, key in enumerate(unique_symbols):\n    category_dict[key] = category_id","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"training_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This function will be used later to make sure our masks are only composed of one single polygon."},{"metadata":{"trusted":true},"cell_type":"code","source":"#Apply larger dilation until the mask is in one block\ndef get_unique_mask(cropped_mask):\n    is_dilation_complete = False\n    cropped_mask = cropped_mask.astype(\"uint8\")\n    \n    #Check if the current mask embeds all features in one \"polygon\"\n    contours= cv2.findContours(cropped_mask,cv2.RETR_TREE,cv2.CHAIN_APPROX_SIMPLE)\n    if len(contours[0])==1:\n        is_dilation_complete = True\n        #just a bit of dilation to make the mask smoother\n        kernel = np.ones((5,5),np.uint8)\n        dilation = cv2.dilate(cropped_mask,kernel,iterations = 1)\n    \n    #Otherwise, let's dilate the mask until it embeds all features\n    kernel_factor = 1\n    while not is_dilation_complete:\n        kernel_size = kernel_factor*5\n        kernel = np.ones((kernel_size,kernel_size),np.uint8)\n        dilation = cv2.dilate(cropped_mask,kernel,iterations = 1)\n\n        contours= cv2.findContours(dilation,cv2.RETR_TREE,cv2.CHAIN_APPROX_SIMPLE)\n        if len(contours[0])==1:\n            is_dilation_complete = True\n\n        kernel_factor+=1\n    \n    #Draw the contours so it fills potential holes in the masks\n    return cv2.drawContours(dilation, contours[0], 0, (255 , 255 , 255),thickness=cv2.FILLED)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This function is the key to prepare the data for a MaskRCNN as we originally did not have any mask provided in the dataset. We basically take the boundaries for each sample, apply an otsu threshold to create the mask, and then dilate the mask until it is composed of a single polygon."},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_mask(img, x, y, width, height):\n\n    #load the cropped area and apply an Otsu threshold\n    cropped_img = np.array(img[y:y+height,x:x+width,:])\n    blurred_img = cv2.GaussianBlur(cropped_img,(5,5),0)\n    img_gray = cv2.cvtColor(blurred_img, cv2.COLOR_BGR2GRAY)\n    ret, otsu = cv2.threshold(img_gray,0,255,cv2.THRESH_BINARY_INV+cv2.THRESH_OTSU)\n    \n    #Place back the cropped area into a mask with the original image size\n    img_height, img_width = img.shape[:2]\n    img_mask = np.full((img_height,img_width),0)\n    img_mask[y:y+height,x:x+width] = otsu\n\n    return img_mask","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"To be able to feed the mask into the MaskRCNN, we need to encode them into RLE."},{"metadata":{"trusted":true},"cell_type":"code","source":"def rle_encoding(x):\n    dots = np.where(x.T.flatten() == 255)[0]\n    run_lengths = []\n    prev = -2\n    for b in dots:\n        if (b>prev+1): run_lengths.extend((b + 1, 0))\n        run_lengths[-1] += 1\n        prev = b\n    return ' '.join([str(x) for x in run_lengths])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Examples"},{"metadata":{"trusted":true},"cell_type":"code","source":"for i in range(5):\n    sample_row = training_df.dropna().sample()\n    random_image = sample_row.image_id.values[0]\n    symbol_metadata_list = np.array(sample_row[\"labels\"].values[0].split(' ')).reshape(-1, 5)\n    \n    x, y, width, height  = [int(x) for x in symbol_metadata_list[12][1:]]\n    \n    print(\"Creating mask for a random character in image {}\".format(random_image))\n    \n    image_filename = \"../../input/kuzushiji-recognition/train_images/{}.jpg\".format(random_image)\n    img = cv2.imread(image_filename,cv2.IMREAD_COLOR)\n    mask = get_mask(img, x, y, width, height)\n\n    cropped_img = img[y:y+height,x:x+width,:]\n    cropped_mask = mask[y:y+height,x:x+width]\n    \n    cropped_single_mask = get_unique_mask(cropped_mask)\n    \n    #just organising everything to display\n    cropped_mask = cropped_mask.copy()\n    masked_img = np.zeros_like(cropped_img)\n    masked_img[cropped_single_mask == 255] = cropped_img[cropped_single_mask == 255]\n    mask[y:y+height,x:x+width] = cropped_single_mask \n    \n    #display all the steps\n    fig, ((ax1, ax2, ax3), (ax4, ax5, ax6)) = plt.subplots(2, 3)\n    ax1.imshow(cropped_img)\n    ax4.imshow(cropped_mask)\n    ax2.imshow(masked_img)\n    ax5.imshow(cropped_single_mask )\n    ax3.imshow(img)\n    ax6.imshow(mask)\n    \n    fig.tight_layout()\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is where the dataframe is completed with all the masks, correponding classes and metadata about the files."},{"metadata":{"trusted":true},"cell_type":"code","source":"training_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Quick line of code to check whether our dataframe contains RLE or not. Just in case you want to reprocess with the original dataframe."},{"metadata":{"trusted":true},"cell_type":"code","source":"has_rle_been_processed = \"EncodedPixels\" in training_df.columns\n\n#formatting the array that were converted to string when saving to .csv\n#also getting rid of images with labels\nif has_rle_been_processed:\n    training_df[\"EncodedPixels\"] = [[str_rle.strip(\" \") for str_rle in rle_list.replace(\"'\",\"\").strip(\"[]\").split(',')] for rle_list in training_df[\"EncodedPixels\"]]\n    training_df[\"CategoryId\"] = [[int(value) for value in str_rle.replace(\" \",\"\").strip(\"[]\").split(',') if value.isdigit()] for str_rle in training_df[\"CategoryId\"]]\n    training_df = training_df.drop(columns=['Unnamed: 0'])\n    training_df = training_df.dropna()\n    training_df = training_df[training_df['CategoryId'].map(lambda category_list: len(category_list)) > 0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"EncodedPixels_list = []\nCategoryId_list = []\nWidth_list = []\nHeight_list = []\n\n#If you use the dataframe provided, it will skip this preprocessing step\nif not has_rle_been_processed:\n    with tqdm(total=len(list(training_df.iterrows()))) as pbar:\n        for idx, document_metadata in training_df.iterrows(): \n            pbar.update(1)\n\n            document_rle_encoding = []\n            document_categories = []\n\n            try:\n                image_filename = \"../../input/kuzushiji-recognition/train_images/{}.jpg\".format(document_metadata[\"image_id\"])\n                img = cv2.imread(image_filename,cv2.IMREAD_COLOR)\n\n                #just getting the width and height of the image for my new metadata dataframe\n                img_width, img_height = Image.open(image_filename).size\n\n                #format the labels information so each item of the list represents a kuzushiji symbol\n                symbol_metadata_list = np.array(document_metadata[\"labels\"].split(' ')).reshape(-1, 5)\n\n                for symbol_metadata in symbol_metadata_list:\n\n                    symbol_unicode = symbol_metadata[0]\n                    if symbol_unicode in kuzushiji_to_detect:\n                        symbol_category = category_dict[symbol_unicode]\n                        document_categories.append(symbol_category)\n\n                        x, y, width, height  = [int(x) for x in symbol_metadata[1:]]\n\n                        mask = get_mask(img, x, y, width, height)\n                        cropped_single_mask = get_unique_mask(mask)\n                        str_rle = rle_encoding(cropped_single_mask)\n\n                        document_rle_encoding.append(str_rle)\n\n            except:\n                print(\"This document had no labels.\")\n\n            EncodedPixels_list.append(document_rle_encoding)\n            CategoryId_list.append(document_categories)\n            Width_list.append(img_width)\n            Height_list.append(img_height)\n\n    training_df[\"EncodedPixels\"] = EncodedPixels_list\n    training_df[\"CategoryId\"] = CategoryId_list\n    training_df[\"Width\"] = Width_list\n    training_df[\"Height\"] = Height_list  ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"training_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def resize_image(image_path):\n    img = cv2.imread(image_path)\n    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n    img = cv2.resize(img, (IMAGE_SIZE, IMAGE_SIZE), interpolation=cv2.INTER_AREA)  \n    return img","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class KuzushijiDataset(utils.Dataset):\n\n    def __init__(self, df):\n        super().__init__(self)\n        \n        # Add classes\n        for i, name in enumerate(kuzushiji_to_detect):\n            self.add_class(\"kuzushiji\", i+1, name)\n        \n        # Add images \n        for i, row in df.iterrows():\n            self.add_image(\"kuzushiji\", \n                           image_id=row.name, \n                           path='../../input/kuzushiji-recognition/train_images/'+str(row.image_id)+\".jpg\", \n                           labels=row['CategoryId'],\n                           annotations=row['EncodedPixels'], \n                           height=row['Height'], width=row['Width'])\n\n    def image_reference(self, image_id):\n        info = self.image_info[image_id]\n        return info['path'], [kuzushiji_to_detect[int(x)] for x in info['labels']]\n        \n    def load_image(self, image_id):\n        return resize_image(self.image_info[image_id]['path'])\n        \n\n    def load_mask(self, image_id):\n        info = self.image_info[image_id]\n                \n        mask = np.zeros((IMAGE_SIZE, IMAGE_SIZE, len(info['annotations'])), dtype=np.uint8)\n        labels = []\n        for m, (annotation, label) in enumerate(zip(info['annotations'], info['labels'])):\n            sub_mask = np.full(info['height']*info['width'], 0, dtype=np.uint8)\n            annotation = [int(x) for x in annotation.split(' ')]\n            \n            for i, start_pixel in enumerate(annotation[::2]):\n                sub_mask[start_pixel: start_pixel+annotation[2*i+1]] = 1\n\n            sub_mask = sub_mask.reshape((info['height'], info['width']), order='F')\n            sub_mask = cv2.resize(sub_mask, (IMAGE_SIZE, IMAGE_SIZE), interpolation=cv2.INTER_NEAREST)\n            \n            mask[:, :, m] = sub_mask\n            labels.append(int(label)+1)\n            \n        return mask, np.array(labels)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"training_percentage = 0.9\n\ntraining_set_size = int(training_percentage*len(training_df))\nvalidation_set_size = int((1-training_percentage)*len(training_df))\n\ntrain_dataset = KuzushijiDataset(training_df[:training_set_size])\ntrain_dataset.prepare()\n\nvalid_dataset = KuzushijiDataset(training_df[training_set_size:training_set_size+validation_set_size])\nvalid_dataset.prepare()\n\nfor i in range(5):\n    image_id = random.choice(train_dataset.image_ids)\n    print(train_dataset.image_reference(image_id))\n    \n    image = train_dataset.load_image(image_id)\n    mask, class_ids = train_dataset.load_mask(image_id)\n    visualize.display_top_masks(image, mask, class_ids, train_dataset.class_names, limit=5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Training the MaskRCNN"},{"metadata":{"trusted":true},"cell_type":"code","source":"LR = 1e-4\nEPOCHS = [1, 2]\n\nimport warnings \nwarnings.filterwarnings(\"ignore\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model = modellib.MaskRCNN(mode='training', config=config, model_dir=ROOT_DIR)\n\n#Using some pretrained weights from a previous kernel, otherwise we can use the COCO weights.\n#We exclude some of the layers as I have just added more classes/symbols.\nmodel.load_weights(\"../../input/maskrcnn-kuzushiji-pretrained-weights-v4/maskuzushiji_pretrained_weights_after_8_epochs.h5\", by_name=True)\n# model.load_weights(COCO_WEIGHTS_PATH, by_name=True, exclude=[\n#     'mrcnn_class_logits', 'mrcnn_bbox_fc', 'mrcnn_bbox', 'mrcnn_mask'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"raw","source":"Train the heads to start"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nmodel.train(train_dataset, valid_dataset,\n            learning_rate=LR,\n            epochs=EPOCHS[0],\n            layers='heads',\n            augmentation=None)\n\nhistory = model.keras_model.history.history","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Train all layers"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nmodel.train(train_dataset, valid_dataset,\n            learning_rate=LR,\n            epochs=EPOCHS[1],\n            layers='all')\n\nnew_history = model.keras_model.history.history\nfor k in new_history: history[k] = history[k] + new_history[k]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"epochs = range(EPOCHS[-1])\n\nplt.figure(figsize=(18, 6))\n\nplt.subplot(131)\nplt.plot(epochs, history['loss'], label=\"train loss\")\nplt.plot(epochs, history['val_loss'], label=\"valid loss\")\nplt.legend()\nplt.subplot(132)\nplt.plot(epochs, history['mrcnn_class_loss'], label=\"train class loss\")\nplt.plot(epochs, history['val_mrcnn_class_loss'], label=\"valid class loss\")\nplt.legend()\nplt.subplot(133)\nplt.plot(epochs, history['mrcnn_mask_loss'], label=\"train mask loss\")\nplt.plot(epochs, history['val_mrcnn_mask_loss'], label=\"valid mask loss\")\nplt.legend()\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"best_epoch = np.argmin(history[\"val_loss\"]) + 1\nprint(\"Best epoch: \", best_epoch)\nprint(\"Valid loss: \", history[\"val_loss\"][best_epoch-1])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Preparing the model for prediction (set to inference mode and load the best weights)"},{"metadata":{"trusted":true},"cell_type":"code","source":"class InferenceConfig(KuzushijiConfig):\n    GPU_COUNT = 1\n    IMAGES_PER_GPU = 1\n\ninference_config = InferenceConfig()\n\nmodel = modellib.MaskRCNN(mode='inference', \n                          config=inference_config,\n                          model_dir=ROOT_DIR)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"glob_list = glob.glob(f'../../working/kuzushiji*/mask_rcnn_kuzushiji_{best_epoch:04d}.h5')\nmodel_path = glob_list[0] if glob_list else ''\nmodel.load_weights(model_path, by_name=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Fix overlapping masks\ndef refine_masks(masks, rois):\n    areas = np.sum(masks.reshape(-1, masks.shape[-1]), axis=0)\n    mask_index = np.argsort(areas)\n    union_mask = np.zeros(masks.shape[:-1], dtype=bool)\n    for m in mask_index:\n        masks[:, :, m] = np.logical_and(masks[:, :, m], np.logical_not(union_mask))\n        union_mask = np.logical_or(masks[:, :, m], union_mask)\n    for m in range(masks.shape[-1]):\n        mask_pos = np.where(masks[:, :, m]==True)\n        if np.any(mask_pos):\n            y1, x1 = np.min(mask_pos, axis=1)\n            y2, x2 = np.max(mask_pos, axis=1)\n            rois[m, :] = [y1, x1, y2, x2]\n    return masks, rois","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Predictions"},{"metadata":{},"cell_type":"markdown","source":"### Displaying some examples and then going through the whole test set"},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_df = pd.read_csv(\"../../input/kuzushiji-recognition/sample_submission.csv\")\nsample_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for i in range(6):\n    image_id = sample_df.sample()[\"image_id\"].values[0]\n    image_path = str('../../input/kuzushiji-recognition/test_images/'+image_id+'.jpg')\n    print(image_path)\n    img = cv2.imread(image_path)\n    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n    \n    result = model.detect([resize_image(image_path)])\n    r = result[0]\n    \n    if r['masks'].size > 0:\n        masks = np.zeros((img.shape[0], img.shape[1], r['masks'].shape[-1]), dtype=np.uint8)\n        for m in range(r['masks'].shape[-1]):\n            masks[:, :, m] = cv2.resize(r['masks'][:, :, m].astype('uint8'), \n                                        (img.shape[1], img.shape[0]), interpolation=cv2.INTER_NEAREST)\n        \n        y_scale = img.shape[0]/IMAGE_SIZE\n        x_scale = img.shape[1]/IMAGE_SIZE\n        rois = (r['rois'] * [y_scale, x_scale, y_scale, x_scale]).astype(int)\n        \n        masks, rois = refine_masks(masks, rois)\n    else:\n        masks, rois = r['masks'], r['rois']\n        \n    visualize.display_instances(img, rois, masks, r['class_ids'], \n                                ['bg']+kuzushiji_to_detect, r['scores'],\n                                title=image_id, figsize=(12, 12))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions = []\nwith tqdm(total=len(sample_df)) as pbar:\n    for i,row in sample_df.iterrows():\n        pbar.update(1)\n        image_id = row[\"image_id\"]\n        image_path = str('../../input/kuzushiji-recognition/test_images/'+image_id+'.jpg')\n        img = cv2.imread(image_path)\n        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n\n        result = model.detect([resize_image(image_path)])\n        r = result[0]\n\n        pred = \"\"\n        if r['masks'].size > 0:\n            masks = np.zeros((img.shape[0], img.shape[1], r['masks'].shape[-1]), dtype=np.uint8)\n            for m in range(r['masks'].shape[-1]):\n                masks[:, :, m] = cv2.resize(r['masks'][:, :, m].astype('uint8'), \n                                            (img.shape[1], img.shape[0]), interpolation=cv2.INTER_NEAREST)\n\n            y_scale = img.shape[0]/IMAGE_SIZE\n            x_scale = img.shape[1]/IMAGE_SIZE\n            rois = (r['rois'] * [y_scale, x_scale, y_scale, x_scale]).astype(int)\n\n            masks, rois = refine_masks(masks, rois)\n\n            pred = \" \".join([\"{} {} {}\".format(kuzushiji_to_detect[class_id],  int((roi[1]+(roi[3]-roi[1])/2)*x_scale), int((roi[0]+(roi[2]-roi[0])/2)*y_scale) ) for class_id, roi in zip(r['class_ids'],r['rois'])] )\n        else:\n            masks, rois = r['masks'], r['rois']\n\n        predictions.append(pred)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Putting back the predictions in the dataframe and generating the submission file. As the model is not great yet and may not give any prediction for a given image, we have to make sure any NaN value is replaced by an empty string for the submission file. I previously had problem to submit the file and I assumed it was coming from there. Fingers crossed!"},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_df[\"labels\"] = predictions\nsample_df.to_csv(\"../submission.csv\",index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### I hope you found this kernel useful! ;)"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}