{"cells":[{"metadata":{},"cell_type":"markdown","source":"# ⚡ Fast ensemble\n-------\nOptimizing your code is usually a good practice, but it is inevitable in a limited environment like Kaggle's Code Competitions. In this Notebook, I implement most of the tips [from this notebook](https://www.kaggle.com/pestipeti/optimization-tips-faster-train-faster-inference). Hopefully, using these, you won't run out of time or memory.\n\n\nI used this inference code, and my submission has finished in 19 minutes.\n\n**Notes**:\n- The duration may vary depending on your model, number of folds, the actual workload on the Kaggle servers, etc.\n- I did not upload my trained weights (I don't want to create high scoring kernel), so the result is an average of 5 Resnet34 models (pretrained weights).\n- Do not forget to enable GPU in your kernel. (I did not test this fork of my code with GPU. It should work.)\n\n## Summary\n- Use script instead of notebook\\*\n- Import only things you need\n- Use logs instead of tqdm\n- Cleanup after usage\n- Load parquet files once\n- Do not load data you don't need\n- Check your dtypes\n- Preprocess your images once\n- Use CUDA for preprocessing (not yet implemented)\n- Do not use albumentation (inference)\n- Only use 3 channels if you really need it\n- Process in batches\n- Optimized TTA (not yet implemented)\n\n\\* *I use notebook for demonstration only. You should copy the code into a script.*\n\n## Explanation\nYou can find more details/explanation about these tricks in this notebook:\n\n[https://www.kaggle.com/pestipeti/optimization-tips-faster-train-faster-inference](https://www.kaggle.com/pestipeti/optimization-tips-faster-train-faster-inference)"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import cv2\nimport numpy as np\nimport pandas as pd\nimport pyarrow.parquet as pq\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\n\nfrom torch.utils.data import Dataset\nfrom torch.utils.data.sampler import SequentialSampler\nfrom torchvision.models import resnet34, densenet121\n\nimport gc","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# ======================\n# Params\n\n# I did not optimize the batch size\n# If the GPU has more memory available\n# you can increase this for faster inference\nBATCH_SIZE = 96\nN_WORKERS = 4\n\nHEIGHT = 137\nWIDTH = 236\nTARGET_SIZE = 128\nPADDING = 8\n\n# Replace these to your values\nMEAN = 0.0778441\nSTD = 0.216016\n\n# You should keep this\nINPUT_PATH = '/kaggle/input/bengaliai-cv19'\n\n# Replace this to your weight dataset\nDATASET_PATH = '/kaggle/input/private-bengali-ai-model-weights'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"NUM_ENSEMBLE = 5\n\n# These are from my experiments (I did not upload the weights)\n# For this demo I used equal weights, but feel free to modify them.\n# Make sure the sum of the labels are equals to 1 (per label)\nENSEMBLES = [\n    {\n        'model': 'resnet',\n        'model_str': 'resnet-34',\n        'model_state_file': DATASET_PATH + '/exp-507--a--f0--resnet-34--swa-4.pt',\n        \n        # Ensemble-item weights\n        'w_grapheme': 1 / NUM_ENSEMBLE,\n        'w_vowel':  1 / NUM_ENSEMBLE,\n        'w_conso':  1 / NUM_ENSEMBLE,\n    },\n    {\n        'model': 'resnet',\n        'model_str': 'resnet-34',\n        'model_state_file': DATASET_PATH + '/exp-560--a--f1--resnet-34--swa-4.pt',\n        'w_grapheme': 1 / NUM_ENSEMBLE,\n        'w_vowel':  1 / NUM_ENSEMBLE,\n        'w_conso':  1 / NUM_ENSEMBLE,\n    },\n    {\n        'model': 'resnet',\n        'model_str': 'resnet-34',\n        'model_state_file': DATASET_PATH + '/exp-588--a--f2--resnet-34--swa-4.pt',\n        'w_grapheme': 1 / NUM_ENSEMBLE,\n        'w_vowel':  1 / NUM_ENSEMBLE,\n        'w_conso':  1 / NUM_ENSEMBLE,\n    },\n    {\n        'model': 'resnet',\n        'model_str': 'resnet-34',\n        'model_state_file': DATASET_PATH + '/exp-589--a--f3--resnet-34--swa-4.pt',\n        'w_grapheme': 1 / NUM_ENSEMBLE,\n        'w_vowel':  1 / NUM_ENSEMBLE,\n        'w_conso':  1 / NUM_ENSEMBLE,\n    },\n    {\n        'model': 'resnet',\n        'model_str': 'resnet-34',\n        'model_state_file': DATASET_PATH + '/exp-590--a--f4--resnet-34--swa-4.pt',\n        'w_grapheme': 1 / NUM_ENSEMBLE,\n        'w_vowel':  1 / NUM_ENSEMBLE,\n        'w_conso':  1 / NUM_ENSEMBLE,\n    },\n]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Preprocessing\nThe preprocessing script below is from Iafoss' kernel:\n\n[https://www.kaggle.com/iafoss/image-preprocessing-128x128](https://www.kaggle.com/iafoss/image-preprocessing-128x128)"},{"metadata":{"trusted":true},"cell_type":"code","source":"def bbox(img):\n    rows = np.any(img, axis=1)\n    cols = np.any(img, axis=0)\n    rmin, rmax = np.where(rows)[0][[0, -1]]\n    cmin, cmax = np.where(cols)[0][[0, -1]]\n    return rmin, rmax, cmin, cmax\n\n\ndef crop_resize(img0, size=TARGET_SIZE, pad=64):\n    # crop a box around pixels large than the threshold\n    # some images contain line at the sides\n    ymin, ymax, xmin, xmax = bbox(img0[5:-5, 5:-5] > 80)\n\n    # cropping may cut too much, so we need to add it back\n    xmin = xmin - 13 if (xmin > 13) else 0\n    ymin = ymin - 10 if (ymin > 10) else 0\n    xmax = xmax + 13 if (xmax < WIDTH - 13) else WIDTH\n    ymax = ymax + 10 if (ymax < HEIGHT - 10) else HEIGHT\n    img = img0[ymin:ymax, xmin:xmax]\n\n    # remove lo intensity pixels as noise\n    img[img < 28] = 0\n    lx, ly = xmax - xmin, ymax - ymin\n    ls = max(lx, ly) + pad\n\n    # make sure that the aspect ratio is kept in rescaling\n    img = np.pad(img, [((ls - ly) // 2,), ((ls - lx) // 2,)], mode='constant')\n\n    return cv2.resize(img, (size, size))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df = pd.read_csv(INPUT_PATH + ('/test.csv'))\nsubmission_df = pd.read_csv(INPUT_PATH + '/sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Loading data"},{"metadata":{"trusted":true},"cell_type":"code","source":"class BengaliParquetDataset(Dataset):\n\n    def __init__(self, num_samples=1):\n        \n        self.num_samples = num_samples\n        self.images = torch.zeros(num_samples, TARGET_SIZE * TARGET_SIZE, dtype=torch.uint8)\n        img_id = 0\n\n        for i in range(4):\n            datafile = INPUT_PATH + '/test_image_data_{}.parquet'.format(i)\n            parq = pq.read_pandas(datafile, columns=[str(x) for x in range(32332)]).to_pandas()\n            parq = 255 - parq.iloc[:, :].values.reshape(-1, HEIGHT, WIDTH).astype(np.uint8)\n            \n            # Not enough memory to do this using a large batch\n            # parq = (parq * (255.0 / parq.max(axis=(1,2), keepdims=True))).astype(np.uint8)\n\n            for idx, image in enumerate(parq):\n                image = (image * (255.0 / image.max())).astype(np.uint8)\n                self.images[img_id, ...] = torch.from_numpy(crop_resize(image, size=TARGET_SIZE, pad=PADDING).reshape(-1).astype(np.uint8))\n                img_id = img_id + 1\n                \n        del parq\n\n    def __len__(self):\n        return len(self.images)\n\n    def __getitem__(self, idx):\n        img = self.images[idx]\n        img = img.view(TARGET_SIZE, TARGET_SIZE)\n        img = img.unsqueeze(0)\n\n        return img, idx","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"bengali_dataset = BengaliParquetDataset(num_samples = test_df.shape[0] // 3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Models"},{"metadata":{"trusted":true},"cell_type":"code","source":"class BengaliResnets(nn.Module):\n\n    def __init__(self, backbone_str='resnet-18'):\n        super().__init__()\n        self.backbone = resnet34(pretrained=False)\n        num_bottleneck_filters = self.backbone.fc.in_features\n        self.head_dropout = 0.1\n        \n        old_conv1 = self.backbone.conv1\n\n        self.backbone.conv1 = nn.Conv2d(1, 64, kernel_size=7, stride=2, padding=3, bias=False)\n        with torch.no_grad():\n            self.backbone.conv1.weight = nn.Parameter(old_conv1.weight.data[:, 0, :, :].unsqueeze(1))        \n\n        self.last = nn.Sequential(\n            nn.BatchNorm2d(num_bottleneck_filters),\n            nn.ReLU()\n        )\n\n        self.fc_graph = nn.Linear(num_bottleneck_filters, 168)\n        self.fc_vowel = nn.Linear(num_bottleneck_filters, 11)\n        self.fc_conso = nn.Linear(num_bottleneck_filters, 7)        \n\n    def forward_backbone(self, x):\n\n        x = self.backbone.conv1(x)\n        x = self.backbone.bn1(x)\n        x = self.backbone.relu(x)\n        x = self.backbone.maxpool(x)\n\n        x = self.backbone.layer1(x)\n        x = self.backbone.layer2(x)\n        x = self.backbone.layer3(x)\n        x = self.backbone.layer4(x)\n\n        x = self.last(x)\n\n        return x\n\n    def forward(self, x):\n        batch_size, C, H, W = x.shape\n        \n        x = (x - MEAN * 255.0) / (STD * 255.0)\n        x = self.forward_backbone(x)\n\n        x = F.adaptive_avg_pool2d(x, 1).reshape(batch_size, -1)\n        x = F.dropout(x, self.head_dropout, self.training)\n\n        fc_graph = self.fc_graph(x)\n        fc_vowel = self.fc_vowel(x)\n        fc_conso = self.fc_conso(x)\n\n        return fc_graph, fc_vowel, fc_conso","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# If you'd like to use different batch size for\n# different size models (tip #12)\ndata_loader_test = torch.utils.data.DataLoader(\n    bengali_dataset,\n    batch_size=BATCH_SIZE,\n    num_workers=N_WORKERS,\n    sampler=SequentialSampler(bengali_dataset),\n    shuffle=False\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Predicting"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Predictions\nsize = submission_df.shape[0] // 3\nresults = {\n    'grapheme_root': np.zeros((len(ENSEMBLES), size, 168), dtype=np.float),\n    'vowel_diacritic': np.zeros((len(ENSEMBLES), size, 11), dtype=np.float),\n    'consonant_diacritic': np.zeros((len(ENSEMBLES), size, 7), dtype=np.float),\n}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for model_idx, ensemble in enumerate(ENSEMBLES):\n    \n    if ensemble['model'].lower() == 'resnet':\n        model = BengaliResnets(backbone_str=ensemble['model_str'].lower())\n    # elif ensemble['model'].lower() == 'densenet':\n    #     model = BengaliDensenets(backbone_str=ensemble['model_str'].lower())\n    else:\n        raise ValueError\n    \n    # Load your model's/fold's weights\n    model_state = None\n    # model_state = torch.load(ensemble['model_state_file'])\n    # model.load_state_dict(model_state['model_state_dict'])\n    model.eval()\n\n    if torch.cuda.is_available():\n        model.cuda()\n    \n    del model_state\n    \n    for batch_idx, data in enumerate(data_loader_test):\n        images, image_idx = data\n\n        if torch.cuda.is_available():\n            images = images.float().cuda()\n        else:\n            images = images.float()\n        \n        with torch.no_grad():\n            out_graph, out_vowel, out_conso = model(images)\n\n        out_graph = F.softmax(out_graph, dim=1).data.cpu().numpy() * ensemble['w_grapheme']\n        out_vowel = F.softmax(out_vowel, dim=1).data.cpu().numpy() * ensemble['w_vowel']\n        out_conso = F.softmax(out_conso, dim=1).data.cpu().numpy() * ensemble['w_conso']\n\n        start = batch_idx * BATCH_SIZE\n        end = min((batch_idx + 1) * BATCH_SIZE, submission_df.shape[0] // 3)\n\n        results['grapheme_root'][model_idx, start:end, :] = out_graph\n        results['vowel_diacritic'][model_idx, start:end, :] = out_vowel\n        results['consonant_diacritic'][model_idx, start:end, :] = out_conso\n        \n        del images\n        del out_graph, out_vowel, out_conso\n            \n    del model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Ensemble"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Clean-up\ndel data_loader_test\ndel bengali_dataset\ndel test_df\n\ngc.collect()\n%reset -f out","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission_df = pd.read_csv(INPUT_PATH + '/sample_submission.csv')\nsubmission_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for l in ['grapheme_root', 'vowel_diacritic', 'consonant_diacritic']:\n    idx = submission_df[submission_df['row_id'].str.contains(l)].index\n    submission_df.iloc[idx, 1] = results[l].sum(axis=0).argmax(axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission_df.to_csv('./submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"-----------"},{"metadata":{},"cell_type":"markdown","source":"**Tanks for reading** If you find this notebook useful, plase vote."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}