{"cells":[{"metadata":{},"cell_type":"markdown","source":"# UNet character detector\n\nThis competition has 2 tasks:\n* detection\n* classification  \n\nHere I do only detection. Classification is made further in the same way as MNIST.  \nFor this purpose I need to predict **not only centers** of characters, but also **bounding boxes**.  \n\nUNet is utilized here to predict two binary masks:  \n1) Bounding boxes  \n2) Centers  "},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"from IPython.display import Image\nImage('https://www.kaggleusercontent.com/kf/20801466/eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0JDLUhTMjU2In0..EfQslrLdboOZrLYYs1alwg.NmjksqZiUNMTQV5nu8Ii-rIdQyOBSD4UM9tHDr5pIPC_4d-tzS0asf--wsPFAXqO0zLVZQ2QMJqOanWzA96h5cTVC-lzdQABAMqr4QHW9SaaX4QjqpvBORJa8fhzwZph3ba_BCG6QmBTNV1ljpOR1Q.fkY_GU1zVC4-Nsvb0_WO7Q/__results___files/__results___1_0.png')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The example prediction is in the picture\n\n*Bounding rectangles can intersect*.  \nThis is why I needed center points.  \nI used smth like voronoi polygons (right picture) to split characters.  \n\nThis gives **~97% f1** score on detection (without classification).  \n\nClassification is made with a separate MNIST-like model.  \n\n### Feel free to ask questions and upvote if you like! :)\n\nReferences:  \n* https://github.com/milesial/Pytorch-UNet\n* https://github.com/usuyama/pytorch-unet"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\n\nprint(os.listdir('/kaggle/input/'))\nprint(os.listdir('/kaggle/input/kuzushiji-recognition/'))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"from PIL import Image, ImageDraw, ImageFont\nimport matplotlib.pyplot as plt\n%matplotlib inline\nfrom tqdm import tqdm_notebook as tqdm\nfrom skimage import io, transform","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Score\nThis is taken from official github https://gist.github.com/SohierDane/a90ef46d79808fe3afc70c80bae45972  \nI modified the score function so that it evaluates **only detection**, not classification  \nOnly one line is changed"},{"metadata":{"trusted":true},"cell_type":"code","source":"import multiprocessing\n\ndef score_page(preds, truth):\n    \"\"\"\n    Scores a single page.\n    Args:\n        preds: prediction string of labels and center points.\n        truth: ground truth string of labels and bounding boxes.\n    Returns:\n        True/false positive and false negative counts for the page\n    \"\"\"\n    tp = 0\n    fp = 0\n    fn = 0\n\n    truth_indices = {\n        'label': 0,\n        'X': 1,\n        'Y': 2,\n        'Width': 3,\n        'Height': 4\n    }\n    preds_indices = {\n        'label': 0,\n        'X': 1,\n        'Y': 2\n    }\n\n    if pd.isna(truth) and pd.isna(preds):\n        return {'tp': tp, 'fp': fp, 'fn': fn}\n\n    if pd.isna(truth):\n        fp += len(preds.split(' ')) // len(preds_indices)\n        return {'tp': tp, 'fp': fp, 'fn': fn}\n\n    if pd.isna(preds):\n        fn += len(truth.split(' ')) // len(truth_indices)\n        return {'tp': tp, 'fp': fp, 'fn': fn}\n\n    truth = truth.split(' ')\n    if len(truth) % len(truth_indices) != 0:\n        raise ValueError('Malformed solution string')\n    truth_label = np.array(truth[truth_indices['label']::len(truth_indices)])\n    truth_xmin = np.array(truth[truth_indices['X']::len(truth_indices)]).astype(float)\n    truth_ymin = np.array(truth[truth_indices['Y']::len(truth_indices)]).astype(float)\n    truth_xmax = truth_xmin + np.array(truth[truth_indices['Width']::len(truth_indices)]).astype(float)\n    truth_ymax = truth_ymin + np.array(truth[truth_indices['Height']::len(truth_indices)]).astype(float)\n\n    preds = preds.split(' ')\n    if len(preds) % len(preds_indices) != 0:\n        raise ValueError('Malformed prediction string' + str(preds))\n    preds_label = np.array(preds[preds_indices['label']::len(preds_indices)])\n    preds_x = np.array(preds[preds_indices['X']::len(preds_indices)]).astype(float)\n    preds_y = np.array(preds[preds_indices['Y']::len(preds_indices)]).astype(float)\n    preds_unused = np.ones(len(preds_label)).astype(bool)\n\n    for xmin, xmax, ymin, ymax, label in zip(truth_xmin, truth_xmax, truth_ymin, truth_ymax, truth_label):\n        # Matching = point inside box & character same & prediction not already used\n        '''\n        Here is my modification:\n        '''\n        matching = (xmin < preds_x) & (xmax > preds_x) & (ymin < preds_y) & (ymax > preds_y) #& (preds_label == label) & preds_unused\n        if matching.sum() == 0:\n            fn += 1\n        else:\n            tp += 1\n            preds_unused[np.argmax(matching)] = False\n    fp += preds_unused.sum()\n    return {'tp': tp, 'fp': fp, 'fn': fn}\n\n\ndef kuzushiji_f1(sub, solution):\n    \"\"\"\n    Calculates the competition metric.\n    Args:\n        sub: submissions, as a Pandas dataframe\n        solution: solution, as a Pandas dataframe\n    Returns:\n        f1 score\n    \"\"\"\n    if not all(sub['image_id'].values == solution['image_id'].values):\n        raise ValueError(\"Submission image id codes don't match solution\")\n\n    pool = multiprocessing.Pool()\n    results = pool.starmap(score_page, zip(sub['labels'].values, solution['labels'].values))\n    pool.close()\n    pool.join()\n\n    tp = sum([x['tp'] for x in results])\n    fp = sum([x['fp'] for x in results])\n    fn = sum([x['fn'] for x in results])\n\n    if (tp + fp) == 0 or (tp + fn) == 0:\n        return 0\n    precision = tp / (tp + fp)\n    recall = tp / (tp + fn)\n    print('precision:', precision, 'recall:', recall)\n    if precision > 0 and recall > 0:\n        f1 = (2 * precision * recall) / (precision + recall)\n    else:\n        f1 = 0\n    return f1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train = pd.read_csv('../input/kuzushiji-recognition/train.csv')\nunicode_map = {codepoint: char for codepoint, char in pd.read_csv('../input/kuzushiji-recognition/unicode_translation.csv').values}\nunicode_vocab = sorted(list(unicode_map))\nunicode2id = {codepoint: i for (i, codepoint) in enumerate(unicode_vocab)}\n\ndf_train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Image transform functions"},{"metadata":{"trusted":true},"cell_type":"code","source":"import cv2\n\nIMG_SIZE = 400\nBATCH_SIZE = 4\n\ndef load_image(path):\n    img = cv2.imread(path)\n    #img = io.imread(path)\n    img = img / 255.\n    return img\n\ndef coords_to_square(coords, shape):\n    new = []\n    w, h = shape[:2]\n    for x, y in coords:\n        if h > w:\n            y = int(np.round(y * IMG_SIZE / h))\n            x = x + (h - w) / 2\n            x = int(np.round(x * IMG_SIZE / h))\n        else:\n            x = int(np.round(x * IMG_SIZE / w))\n            y = y + (w - h) / 2\n            y = int(np.round(y * IMG_SIZE / w))\n        new.append([x, y])\n    return np.array(new)\n\ndef to_square(img, img_size=IMG_SIZE):\n    three_d = len(img.shape) == 3\n    if three_d:\n        w, h, c = img.shape\n    else:\n        w, h = img.shape\n        c = 1\n    if w > h:\n        h = int(h * img_size / w)\n        w = img_size\n    else:\n        w = int(w * img_size / h)\n        h = img_size\n    img = cv2.resize(img, (h, w), interpolation=cv2.INTER_NEAREST).reshape([w, h, c])\n    margin_w = (img_size - w) // 2\n    margin_h = (img_size - h) // 2\n    new_img = np.zeros((img_size, img_size, c))\n    new_img[margin_w: margin_w + w, margin_h: margin_h + h, :] = img\n    if not three_d:\n        new_img = new_img.reshape([img_size, img_size])\n    return new_img.astype('float32')\n\ndef unsquare(img, width, height, coords=None):\n    if coords is None:\n        if width > height:\n            w = IMG_SIZE\n            h = int(height * IMG_SIZE / width)\n        else:\n            h = IMG_SIZE\n            w = int(width * IMG_SIZE / height)\n        margin_w = (IMG_SIZE - w) // 2\n        margin_h = (IMG_SIZE - h) // 2\n        img = img[margin_w: margin_w + w, margin_h: margin_h + h]\n        img = cv2.resize(img, (height, width))\n    else:\n        [x1, y1], [x2, y2] = coords\n        [sx1, sy1], [sx2, sy2] = coords_to_square(coords, [width, height])\n        img = cv2.resize(img[sx1: sx2, sy1: sy2], (y2 - y1, x2 - x1))\n    return img\n\ndef get_mask(img, labels):\n    mask = np.zeros((img.shape[0], img.shape[1], 2), dtype='float32')\n    if isinstance(labels, str):\n        labels = np.array(labels.split(' ')).reshape(-1, 5)\n        for char, x, y, w, h in labels:\n            x, y, w, h = int(x), int(y), int(w), int(h)\n            if x + w >= img.shape[1] or y + h >= img.shape[0]:\n                continue\n            mask[y: y + h, x: x + w, 0] = 1\n            radius = 6\n            mask[y + h // 2 - radius: y + h // 2 + radius + 1, x + w // 2 - radius: x + w // 2 + radius + 1, 1] = 1\n    return mask\n\ndef preprocess(img, width, height):\n    skip = 8\n    if width > height:\n        w = IMG_SIZE\n        h = int(height * IMG_SIZE / width)\n    else:\n        h = IMG_SIZE\n        w = int(width * IMG_SIZE / height)\n    margin_w = (IMG_SIZE - w) // 2\n    margin_h = (IMG_SIZE - h) // 2\n    sl_x = slice(margin_w, margin_w + w)\n    sl_y = slice(margin_h, margin_h + h)\n    stat = img[margin_w:margin_w + w:skip, margin_h:margin_h + h:skip].reshape([-1, 3])\n    img[sl_x, sl_y] = img[sl_x, sl_y] - np.median(stat, 0)\n    img[sl_x, sl_y] = img[sl_x, sl_y] / np.std(stat, 0)\n    return img","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"np.random.seed(1337)\n\nfor i in range(4):\n    idx, labels = df_train.values[np.random.randint(len(df_train))]\n    img = load_image('../input/kuzushiji-recognition/train_images/{}.jpg'.format(idx))\n    width, height, _ = img.shape\n    mask = get_mask(img, labels)\n    img_square = preprocess(to_square(img), img.shape[0], img.shape[1])\n    img_unsq = unsquare(img_square, width, height)\n    \n    print(img.shape, mask.shape, img_square.shape, img_unsq.shape)\n    \n    fig, axs = plt.subplots(1, 5, figsize=(30, 8))\n    axs[0].imshow(img)\n    axs[1].imshow(mask[:, :, 0], interpolation='bilinear')\n    axs[2].imshow(mask[:, :, 1], interpolation='bilinear')\n    pos = axs[3].imshow(img_square.sum(2))\n    fig.colorbar(pos, ax=axs[3])\n    axs[4].imshow(img_unsq)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch\nfrom torch.utils.data import Dataset, DataLoader\nfrom torchvision import transforms, utils\n\n# Ignore warnings\nimport warnings\nwarnings.filterwarnings(\"ignore\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data generator for torch models"},{"metadata":{"trusted":true},"cell_type":"code","source":"class PapirusDataset(Dataset):\n    \"\"\"Papirus dataset.\"\"\"\n\n    def __init__(self, dataframe, root_dir, training=True, transform=None):\n        self.df = dataframe\n        self.root_dir = root_dir\n        self.transform = transform\n        self.training = training\n\n    def __len__(self):\n        return len(self.df)\n\n    def __getitem__(self, idx):\n        if torch.is_tensor(idx):\n            idx = idx.tolist()\n        \n        idx, labels = self.df.values[idx]\n        img_name = self.root_dir.format(idx)\n        \n        image = load_image(img_name)\n        if self.training:\n            mask = get_mask(image, labels)\n        shape = np.array(image.shape)[:2]\n        \n        image = to_square(image)\n        image = preprocess(image, shape[0], shape[1])\n        if self.training:\n            mask = to_square(mask)\n        \n        image = np.rollaxis(image, 2, 0)\n        if self.training:\n            mask = np.rollaxis(mask, 2, 0)\n        else:\n            mask = 0\n        \n        sample = [image, mask, shape]\n        \n        if self.transform:\n            sample = self.transform(sample)\n        \n        return sample","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for batch in DataLoader(PapirusDataset(df_train, '../input/kuzushiji-recognition/train_images/{}.jpg'),\n                        batch_size=BATCH_SIZE,\n                        shuffle=True,\n                        num_workers=4):\n    print(batch[0].shape)\n    plt.imshow(batch[0][0].sum(0))\n    break","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Define model"},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torchvision import models\n\nclass double_conv(nn.Module):\n    '''(conv => BN => ReLU) * 2'''\n    def __init__(self, in_ch, out_ch):\n        super(double_conv, self).__init__()\n        self.conv = nn.Sequential(\n            nn.Conv2d(in_ch, out_ch, 3, padding=1),\n            nn.BatchNorm2d(out_ch),\n            nn.ReLU(inplace=True),\n            nn.Conv2d(out_ch, out_ch, 3, padding=1),\n            nn.BatchNorm2d(out_ch),\n            nn.ReLU(inplace=True)\n        )\n\n    def forward(self, x):\n        x = self.conv(x)\n        return x\n\n\nclass inconv(nn.Module):\n    def __init__(self, in_ch, out_ch):\n        super(inconv, self).__init__()\n        self.conv = double_conv(in_ch, out_ch)\n\n    def forward(self, x):\n        x = self.conv(x)\n        return x\n\n\nclass down(nn.Module):\n    def __init__(self, in_ch, out_ch):\n        super(down, self).__init__()\n        self.mpconv = nn.Sequential(\n            nn.MaxPool2d(2),\n            double_conv(in_ch, out_ch)\n        )\n\n    def forward(self, x):\n        x = self.mpconv(x)\n        return x\n\n\nclass up(nn.Module):\n    def __init__(self, in_ch, out_ch, bilinear=True):\n        super(up, self).__init__()\n\n        #  would be a nice idea if the upsampling could be learned too,\n        #  but my machine do not have enough memory to handle all those weights\n        if bilinear:\n            self.up = nn.Upsample(scale_factor=2, mode='bilinear', align_corners=True)\n        else:\n            self.up = nn.ConvTranspose2d(in_ch//2, in_ch//2, 2, stride=2)\n\n        self.conv = double_conv(in_ch, out_ch)\n\n    def forward(self, x1, x2):\n        x1 = self.up(x1)\n        \n        # input is CHW\n        diffY = x2.size()[2] - x1.size()[2]\n        diffX = x2.size()[3] - x1.size()[3]\n\n        x1 = F.pad(x1, (diffX // 2, diffX - diffX//2,\n                        diffY // 2, diffY - diffY//2))\n        \n        # for padding issues, see \n        # https://github.com/HaiyongJiang/U-Net-Pytorch-Unstructured-Buggy/commit/0e854509c2cea854e247a9c615f175f76fbb2e3a\n        # https://github.com/xiaopeng-liao/Pytorch-UNet/commit/8ebac70e633bac59fc22bb5195e513d5832fb3bd\n\n        x = torch.cat([x2, x1], dim=1)\n        x = self.conv(x)\n        return x\n\n\nclass outconv(nn.Module):\n    def __init__(self, in_ch, out_ch):\n        super(outconv, self).__init__()\n        self.conv = nn.Conv2d(in_ch, out_ch, 1)\n\n    def forward(self, x):\n        x = self.conv(x)\n        return x\n\nclass UNet(nn.Module):\n    '''https://github.com/milesial/Pytorch-UNet'''\n    def __init__(self, n_classes):\n        super(UNet, self).__init__()\n        self.inc = inconv(3, 64)\n        self.down1 = down(64, 128)\n        self.down2 = down(128, 256)\n        self.down3 = down(256, 512)\n        self.down4 = down(512, 512)\n        self.up1 = up(1024, 256)\n        self.up2 = up(512, 128)\n        self.up3 = up(256, 64)\n        self.up4 = up(128, 64)\n        self.outc = outconv(64, n_classes)\n\n    def forward(self, x):\n        x1 = self.inc(x)\n        x2 = self.down1(x1)\n        x3 = self.down2(x2)\n        x4 = self.down3(x3)\n        x5 = self.down4(x4)\n        x = self.up1(x5, x4)\n        x = self.up2(x, x3)\n        x = self.up3(x, x2)\n        x = self.up4(x, x1)\n        x = self.outc(x)\n        return x\n\ndef convrelu(in_channels, out_channels, kernel, padding):\n    return nn.Sequential(\n        nn.Conv2d(in_channels, out_channels, kernel, padding=padding),\n        nn.ReLU(inplace=True),\n    )\n\n\nclass ResNetUNet(nn.Module):\n    '''https://github.com/usuyama/pytorch-unet'''\n    def __init__(self, n_classes):\n        super().__init__()\n\n        self.base_model = models.resnet18(pretrained=True)\n        self.base_layers = list(self.base_model.children())\n\n        self.layer0 = nn.Sequential(*self.base_layers[:3]) # size=(N, 64, x.H/2, x.W/2)\n        self.layer0_1x1 = convrelu(64, 64, 1, 0)\n        self.layer1 = nn.Sequential(*self.base_layers[3:5]) # size=(N, 64, x.H/4, x.W/4)\n        self.layer1_1x1 = convrelu(64, 64, 1, 0)\n        self.layer2 = self.base_layers[5]  # size=(N, 128, x.H/8, x.W/8)\n        self.layer2_1x1 = convrelu(128, 128, 1, 0)\n        self.layer3 = self.base_layers[6]  # size=(N, 256, x.H/16, x.W/16)\n        self.layer3_1x1 = convrelu(256, 256, 1, 0)\n        self.layer4 = self.base_layers[7]  # size=(N, 512, x.H/32, x.W/32)\n        self.layer4_1x1 = convrelu(512, 512, 1, 0)\n\n        self.upsample = nn.Upsample(scale_factor=2, mode='bilinear', align_corners=True)\n\n        self.conv_up3 = convrelu(256 + 512, 512, 3, 1)\n        self.conv_up2 = convrelu(128 + 512, 256, 3, 1)\n        self.conv_up1 = convrelu(64 + 256, 256, 3, 1)\n        self.conv_up0 = convrelu(64 + 256, 128, 3, 1)\n\n        self.conv_original_size0 = convrelu(3, 64, 3, 1)\n        self.conv_original_size1 = convrelu(64, 64, 3, 1)\n        self.conv_original_size2 = convrelu(64 + 128, 64, 3, 1)\n\n        self.conv_last = nn.Conv2d(64, n_classes, 1)\n\n    def forward(self, input):\n        x_original = self.conv_original_size0(input)\n        x_original = self.conv_original_size1(x_original)\n\n        layer0 = self.layer0(input)\n        layer1 = self.layer1(layer0)\n        layer2 = self.layer2(layer1)\n        layer3 = self.layer3(layer2)\n        layer4 = self.layer4(layer3)\n\n        layer4 = self.layer4_1x1(layer4)\n        x = self.upsample(layer4)\n        layer3 = self.layer3_1x1(layer3)\n        x = torch.cat([x, layer3], dim=1)\n        x = self.conv_up3(x)\n\n        x = self.upsample(x)\n        layer2 = self.layer2_1x1(layer2)\n        x = torch.cat([x, layer2], dim=1)\n        x = self.conv_up2(x)\n\n        x = self.upsample(x)\n        layer1 = self.layer1_1x1(layer1)\n        x = torch.cat([x, layer1], dim=1)\n        x = self.conv_up1(x)\n\n        x = self.upsample(x)\n        layer0 = self.layer0_1x1(layer0)\n        x = torch.cat([x, layer0], dim=1)\n        x = self.conv_up0(x)\n\n        x = self.upsample(x)\n        x = torch.cat([x, x_original], dim=1)\n        x = self.conv_original_size2(x)\n\n        out = self.conv_last(x)\n\n        return out\n\nclass MyUNet(nn.Module):\n    '''Mixture of previous classes'''\n    def __init__(self, n_classes):\n        super(MyUNet, self).__init__()\n        self.base_model = models.resnet18(pretrained=True)\n        self.base_layers = list(self.base_model.children())\n        \n        self.conv_orig0 = inconv(3, 32)\n        self.conv_orig1 = double_conv(32, 64)\n        \n        self.down0 = nn.Sequential(*self.base_layers[:3]) # size=(N, 64, x.H/2, x.W/2)\n        self.down1 = nn.Sequential(*self.base_layers[3:5]) # size=(N, 64, x.H/4, x.W/4)\n        self.down2 = self.base_layers[5]  # size=(N, 128, x.H/8, x.W/8)\n        self.down3 = self.base_layers[6]  # size=(N, 256, x.H/16, x.W/16)\n        self.down4 = self.base_layers[7]  # size=(N, 512, x.H/32, x.W/32)\n        self.up1 = up(512 + 256, 256)\n        self.up2 = up(256 + 128, 128)\n        self.up3 = up(128 + 64, 64)\n        self.up4 = up(64 + 64, 64)\n        self.up5 = up(64 + 64, 64)\n        #self.upsample = nn.Upsample(scale_factor=2, mode='bilinear', align_corners=True)\n        self.outc = outconv(64, n_classes)\n\n    def forward(self, x):\n        x_orig = self.conv_orig0(x)\n        x_orig = self.conv_orig1(x_orig)\n        \n        x1 = self.down0(x)\n        x2 = self.down1(x1)\n        x3 = self.down2(x2)\n        x4 = self.down3(x3)\n        x5 = self.down4(x4)\n        x = self.up1(x5, x4)\n        x = self.up2(x, x3)\n        x = self.up3(x, x2)\n        x = self.up4(x, x1)\n        x = self.up5(x, x_orig)\n        #x = self.upsample(x)\n        x = self.outc(x)\n        return x","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def dice_loss(pred, target, smooth = 1.):\n    pred = pred.contiguous()\n    target = target.contiguous()    \n\n    intersection = (pred * target).sum(dim=2).sum(dim=2)\n    \n    loss = (1 - ((2. * intersection + smooth) / (pred.sum(dim=2).sum(dim=2) + target.sum(dim=2).sum(dim=2) + smooth)))\n    \n    return loss.mean()\n\ndef calc_loss(pred, target, metrics, bce_weight=0.5):\n    bce = F.binary_cross_entropy_with_logits(pred, target)\n\n    pred = F.sigmoid(pred)\n    dice = dice_loss(pred, target)\n\n    loss = bce * bce_weight + dice * (1 - bce_weight)\n\n    metrics['bce'] = metrics.get('bce', []) + [bce.data.cpu().numpy()]\n    metrics['dice'] = metrics.get('dice', []) + [dice.data.cpu().numpy()]\n    metrics['loss'] = metrics.get('loss', []) + [loss.data.cpu().numpy()]\n\n    return loss","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Train"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\nfrom sklearn.model_selection import train_test_split\nfrom collections import defaultdict\nimport gc\n\ndevice = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\nprint(device)\n\ntorch.cuda.empty_cache()\ngc.collect()\n\nmodel = MyUNet(n_classes=2)\nmodel = model.to(device)\n\ntorch.cuda.empty_cache()\ngc.collect()\n\noptimizer = torch.optim.Adam(filter(lambda p: p.requires_grad, model.parameters()), lr=1e-4)\n\nnum_epochs = 30\n\nxy_train, xy_dev = train_test_split(df_train, test_size=0.01)\n\ndataloaders = {'train': DataLoader(PapirusDataset(xy_train, '../input/kuzushiji-recognition/train_images/{}.jpg'),\n                        batch_size=BATCH_SIZE,\n                        shuffle=True,\n                        num_workers=8),\n              'val': DataLoader(PapirusDataset(xy_dev, '../input/kuzushiji-recognition/train_images/{}.jpg'),\n                        batch_size=BATCH_SIZE,\n                        shuffle=False,\n                        num_workers=8)}\n\nhistory = {'train': [], 'val': []}\n\nfor epoch in range(num_epochs):\n    print('Epoch {}/{}'.format(epoch + 1, num_epochs))\n    \n    for phase in ['train', 'val']:\n        if phase == 'train':\n            model.train()  # Set model to training mode\n        else:\n            model.eval()   # Set model to evaluate mode\n        metrics = {}\n        for inputs, labels, shape in dataloaders[phase]:\n            inputs = inputs.to(device)\n            labels = labels.to(device)\n            \n            optimizer.zero_grad()\n            outputs = model(inputs)\n            loss = calc_loss(outputs, labels, metrics)\n            # backward + optimize only if in training phase\n            if phase == 'train':\n                loss.backward()\n                optimizer.step()\n        history[phase].append({k: np.mean(metrics[k]) for k in metrics})\n        print(phase, history[phase][-1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"torch.save(model.state_dict(), './model')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history_tr = pd.DataFrame({'train_' + k: [metrics[k] for metrics in history['train']] for k in history['train'][0]})\nhistory_val = pd.DataFrame({'val_' + k: [metrics[k] for metrics in history['val']] for k in history['val'][0]})\ndf_history = pd.concat([history_tr, history_val], 1)\nfig = plt.figure(figsize=(12, 6))\nax = plt.axes()\ndf_history.plot(ax=ax)\ndf_history","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Make predictions on dev set"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.cluster import KMeans\n\ndef get_centers(mask):\n    contours, _ = cv2.findContours(mask, cv2.RETR_TREE, cv2.CHAIN_APPROX_NONE)\n    centers = []\n    for cnt in contours:\n        M = cv2.moments(cnt)\n        if M['m00'] > 0:\n            cy = M['m10'] / M['m00']\n            cx = M['m01'] / M['m00']\n        else:\n            cy, cx = cnt[0][0]\n        cx = int(np.round(cx))\n        cy = int(np.round(cy))\n        centers.append([cx, cy])\n    centers = np.array(centers)\n    return centers\n\ndef get_labels(centers, shape):\n    if len(centers) == 0:\n        return\n    kmeans = KMeans(len(centers), init=centers)\n    kmeans.fit(centers)\n    coords = []\n    mlt = 2\n    for i in range(0, shape[0], mlt):\n        coords.append([])\n        for j in range(0, shape[1], mlt):\n            coords[-1].append([i, j])\n    coords = np.array(coords).reshape([-1, 2])\n    preds = kmeans.predict(coords)\n    preds = preds.reshape([shape[0] // mlt, shape[1] // mlt])\n    labels = np.zeros(shape, dtype='int')\n    for k in range(mlt):\n        labels[k::mlt, k::mlt] = preds\n    return labels\n\ndef get_voronoi(centers, mask):\n    labels = get_labels(centers, mask.shape)\n    colors = np.random.uniform(0, 1, size=[len(centers), 3])\n    voronoi = colors[labels]\n    voronoi *= mask[:, :, None]\n    return voronoi\n\ndef get_rectangles(centers, mask):\n    mask_sq = to_square(mask)\n    centers_sq = coords_to_square(centers, mask.shape)\n    labels_sq = get_labels(centers_sq, mask_sq.shape)\n    rects = [None for _ in centers]\n    valid_centers = []\n    for i, (xc, yc) in enumerate(centers):\n        msk = (labels_sq == i).astype('float') * mask_sq / mask_sq.max()\n        # crop msk\n        max_size = 400\n        x1 = max(0, int(np.round(xc - max_size // 2)))\n        y1 = max(0, int(np.round(yc - max_size // 2)))\n        x2 = min(mask.shape[0], int(np.round(xc + max_size // 2)))\n        y2 = min(mask.shape[1], int(np.round(yc + max_size // 2)))\n        msk = unsquare(msk, mask.shape[0], mask.shape[1], coords=[[x1,y1], [x2, y2]])\n        msk = cv2.inRange(msk, 0.5, 10000)\n        contours, _ = cv2.findContours(msk, cv2.RETR_TREE, cv2.CHAIN_APPROX_NONE)\n        for cnt in contours:\n            y, x, h, w = cv2.boundingRect(cnt)\n            x += x1\n            y += y1\n            if xc >= x and xc <= x + w and yc >= y and yc <= y + h:\n                rects[i] = [x, y, w, h]\n                if cv2.contourArea(cnt) <= h * w * 0.66:\n                    rad_x = min(xc - x, x + w - xc)\n                    rad_y = min(yc - y, y + h - yc)\n                    rects[i] = [int(np.round(xc - rad_x)), y, int(np.round(2 * rad_x)), h]\n                break\n        if rects[i] is not None:\n            valid_centers.append([xc, yc])\n    return np.array([r for r in rects if r is not None]), np.array(valid_centers)\n\ndef draw_rectangles(img, rects, centers, fill_rect=[1, 0, 0], fill_cent=[1, 0, 0], fill_all=False):\n    new = np.array(img)\n    for x, y, w, h in rects:\n        for shift in range(4):\n            try:\n                if fill_all:\n                    new[x: x + w, y: y + h] = fill_rect\n                else:\n                    new[x: x + w, y + shift] = fill_rect\n                    new[x: x + w, y + h - shift] = fill_rect\n                    new[x + shift, y: y + h] = fill_rect\n                    new[x + w - shift, y: y + h] = fill_rect\n            except:\n                pass\n    for x, y in centers:\n        r = 15\n        new[x - r: x + r, y - r: y + r] = fill_cent\n    return new\n\ndef add_skipped(mask, boxes, centers):\n    avg_w = np.mean([b[2] for b in boxes])\n    avg_area = np.mean([b[2] * b[3] for b in boxes])\n    new_centers, new_boxes = [], []\n    mask_c = draw_rectangles(mask, boxes, [], 0, fill_all=True)\n    contours, _ = cv2.findContours(mask_c, cv2.RETR_TREE, cv2.CHAIN_APPROX_NONE)\n    for cnt in contours:\n        y, x, h, w = cv2.boundingRect(cnt)\n        found = False\n        for xc, yc in centers:\n            if xc >= x and xc <= x + w and yc >= y and yc <= y + h:\n                found = True\n                break\n        if not found and (w * h > avg_area * 0.66 or w > avg_w * 1.5):\n            new_centers.append([x + w // 2, y + h // 2])\n            new_boxes.append([x, y, w, h])\n    if len(new_centers) > 0:\n        boxes = np.concatenate([boxes, new_boxes], 0)\n        centers = np.concatenate([centers, new_centers], 0)\n    return boxes, centers","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"val_pred_boxes = []\nval_pred_centers = []\n\ndataloaders = {'val': DataLoader(PapirusDataset(xy_dev, '../input/kuzushiji-recognition/train_images/{}.jpg'),\n                        batch_size=4,\n                        shuffle=False,\n                        num_workers=8)}\ngc.collect()\n\nfor inputs, labels, shape in tqdm(dataloaders['val']):\n    inputs = inputs.to(device)\n    outputs = model(inputs)\n    \n    for i in range(len(inputs.data.cpu().numpy())):\n        img = inputs.data.cpu().numpy()[i]\n        mask = outputs.data.cpu().numpy()[i]\n        shp = shape.data.cpu().numpy()[i]\n        \n        binary0 = cv2.inRange(unsquare(mask[0], shp[0], shp[1]), 0.0, 10000)\n        binary1 = cv2.inRange(unsquare(mask[1], shp[0], shp[1]), 0.0, 10000)\n        \n        centers0 = get_centers(binary1)\n        rects, centers = get_rectangles(centers0, binary0)\n        rects, centers = add_skipped(binary0, rects, centers)\n        \n        cnts = []\n        for x, y in centers:\n            cnts += ['unk', str(y), str(x)]\n        cnts = None if len(cnts) == 0 else ' '.join(cnts)\n        val_pred_centers.append(cnts)\n        \n        boxes = []\n        for x, y, w, h in rects:\n            boxes += ['unk', str(y), str(x), str(h), str(w)]\n        boxes = None if len(boxes) == 0 else ' '.join(boxes)\n        val_pred_boxes.append(boxes)\n        \n        if i == 0 and len(centers) != 0:\n            img = unsquare(np.rollaxis(img, 0, 3), shp[0], shp[1])\n            print(len(rects), 'boxes of', len(centers0), 'centers')\n            img_min = img.min(0, keepdims=True).min(1, keepdims=True)\n            img_max = img.max(0, keepdims=True).max(1, keepdims=True)\n            drawn = draw_rectangles((img - img_min) / (img_max - img_min), rects, centers0)\n            fig, axs = plt.subplots(1, 4, figsize=(30, 8))\n            axs[0].imshow(drawn, interpolation='bilinear')\n            axs[1].imshow(binary0, interpolation='bilinear')\n            axs[2].imshow(binary1, interpolation='bilinear')\n            centers_sq = coords_to_square(centers, binary0.shape)\n            axs[3].imshow(get_voronoi(centers_sq, cv2.inRange(mask[0], 0.0, 10000) / 255))\n\nval_predictions = pd.DataFrame({'image_id': xy_dev['image_id'], 'labels': val_pred_centers})\nval_predictions_box = pd.DataFrame({'image_id': xy_dev['image_id'], 'labels': val_pred_boxes})\n\nkuzushiji_f1(val_predictions, xy_dev)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"val_predictions.to_csv('dev_centers.csv', index=False)\nval_predictions_box.to_csv('dev_boxes.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`val_predictions` I use to evaluate the detection  \n`val_predictions_box` I use to extract charachters and further classify them  \n\nI use separate kernel to make predictions on the test set, since it is large. The code is the same, and it uses trained model from this kernel  \nPrediction of the full test set takes approx. 2 hours without gpu."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}