{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":" # Testing and Creating a DataLoader\nWith new GPU limits, every single second saved is important. Dataloading take a lot of time per epoch so here in this notebook I have decided to test them out.\nIn this notebook I will try various formats of dataloaders and try to find out which is the fastest. In particular, I will compare the time taken to run a certain number of epcohs for the following four kinds of dataloader:\n1. Load Original Images and Resize + Create masks on spot\n2. Load Resized Images + Create masks on spot\n3. Load Resized Images in numpy format + Load masks in numpy format\n4. Load both images and maps from RAM (unfortunately, with the resources I have and the resources kaggle kernels provide, it's not possible to load both images and masks in RAM unless you reduce the size a lot)\n\nThere are four ways to load an image:\n1. Load large image then resize\n2. Load resized image\n3. Load resized image saved as numpy array\n4. Load resized image from RAM\n\nThere are three ways to load the mask:\n1. Create mask on spot them resize\n2. Load mask from numpy array and don't resize\n3. Load mask from the RAM\n\nBy various combinations of the above, twelve dataloaders are possible. But, I am just testing four dataloaders which I believe are the most important."},{"metadata":{},"cell_type":"markdown","source":"# Prepare Data\nFirst, we need to resize and save images as both .jpg and .npy array. We also create and save the mask on disk as .npy array. \nWe have limited disk space and RAM (for RAM loader) so let's just work with 1k images. "},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport cv2\nimport numpy as np\nimport pandas as pd\nfrom tqdm import tqdm\nimport albumentations as albu\n\ndef get_img(image_path):\n    \"\"\"Load image from disk\"\"\"\n    img = cv2.imread(image_path)\n    return img\n\ndef rle_decode(mask_rle: str = \"\", shape: tuple = (1400, 2100)):\n    \"\"\"Source: https://www.kaggle.com/artgor/segmentation-in-pytorch-using-convenient-tools\"\"\"\n    s = mask_rle.split()\n    starts, lengths = [np.asarray(x, dtype=int) for x in (s[0:][::2], s[1:][::2])]\n    starts -= 1\n    ends = starts + lengths\n    img = np.zeros(shape[0] * shape[1], dtype=np.uint8)\n    for lo, hi in zip(starts, ends):\n        img[lo:hi] = 1\n    return img.reshape(shape, order=\"F\")\n\ndef make_mask(df: pd.DataFrame, image_name: str='img.jpg', shape: tuple = (1400, 2100)):\n    \"\"\"Source: https://www.kaggle.com/artgor/segmentation-in-pytorch-using-convenient-tools\"\"\"\n    encoded_masks = df.loc[df['im_id'] == image_name, 'EncodedPixels']\n    masks = np.zeros((shape[0], shape[1], 4), dtype=np.float32)\n    for idx, label in enumerate(encoded_masks.values):\n        if label is not np.nan:\n            mask = rle_decode(label)\n            masks[:, :, idx] = mask         \n    return masks\n\n\n#Load train.csv to make mask\ntrain = pd.read_csv(f\"../input/understanding_cloud_organization/train.csv\")\ntrain[\"label\"] = train[\"Image_Label\"].apply(lambda x: x.split(\"_\")[1])\ntrain[\"im_id\"] = train[\"Image_Label\"].apply(lambda x: x.split(\"_\")[0])\n\n\nN = 1000 #number of images\ndir_ip = '../input/understanding_cloud_organization/train_images'\ndir_op = 'img_resized'\ndir_op_mask = 'mask'\ndir_op_npy = 'img_resized_npy'\n\nfor d in [dir_op, dir_op_mask, dir_op_npy]:\n    if not os.path.exists(d):\n        os.makedirs(d)\n\ntfms = albu.Compose([albu.Resize(320, 640)]) #To resize\nbar = tqdm(os.listdir(dir_ip)[:N], postfix={\"file\":\"none\"})\n\nfor file in bar:\n    bar.set_postfix(ordered_dict={\"file\":file})    \n    path = os.path.join(dir_ip, file)\n    img = get_img(path)    \n    mask = make_mask(train, file) \n    tfmed = tfms(image=img, mask=mask)\n    img = tfmed['image']\n    mask = tfmed['mask']\n    cv2.imwrite(os.path.join(dir_op, file), img)\n    np.save(os.path.join(dir_op_mask, file), mask)\n    np.save(os.path.join(dir_op_npy, file), cv2.cvtColor(img, cv2.COLOR_BGR2RGB))     ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"# Make DataLoaders"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Some utility functions\n\nimport matplotlib.pyplot as plt\n\ndef visualize(image, mask, original_image=None, original_mask=None, gray=True):\n    \"\"\"Source: https://www.kaggle.com/artgor/segmentation-in-pytorch-using-convenient-tools\"\"\"\n    fontsize = 14\n    class_dict = {0: \"Fish\", 1: \"Flower\", 2: \"Gravel\", 3: \"Sugar\"}    \n    if original_image is None and original_mask is None:\n        f, ax = plt.subplots(1, 5, figsize=(24, 24))\n\n        if gray:\n            ax[0].imshow(image, cmap='gray')\n        else:    \n            ax[0].imshow(image)\n        for i in range(4):\n            ax[i + 1].imshow(mask[:, :, i])\n            ax[i + 1].set_title(f\"Mask {class_dict[i]}\", fontsize=fontsize)\n    else:\n        f, ax = plt.subplots(2, 5, figsize=(24, 12))\n\n        ax[0, 0].imshow(original_image)\n        ax[0, 0].set_title(\"Original image\", fontsize=fontsize)\n\n        for i in range(4):\n            ax[0, i + 1].imshow(original_mask[:, :, i])\n            ax[0, i + 1].set_title(f\"Original mask {class_dict[i]}\", fontsize=fontsize)\n\n        ax[1, 0].imshow(image)\n        ax[1, 0].set_title(\"Transformed image\", fontsize=fontsize)\n\n        for i in range(4):\n            ax[1, i + 1].imshow(mask[:, :, i])\n            ax[1, i + 1].set_title(\n                f\"Transformed mask {class_dict[i]}\", fontsize=fontsize\n            )\n\n            \ndef get_img(name, image_dir='dir_op_npy', npy=False):\n    if npy:\n        return np.load(os.path.join(image_dir, name+'.npy'))\n    img = cv2.imread(os.path.join(image_dir, name))\n    return cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n        \n\ndef make_mask(image_name, mask_dir=False, df=False, shape: tuple = (1400, 2100)):\n    if mask_dir: #Load numpy mask\n        return np.load(os.path.join(mask_dir, image_name+'.npy'))  \n    #Make mask\n    encoded_masks = df.loc[df['im_id'] == image_name, 'EncodedPixels']\n    masks = np.zeros((shape[0], shape[1], 4), dtype=np.float32)\n    for idx, label in enumerate(encoded_masks.values):\n        if label is not np.nan:\n            mask = rle_decode(label)\n            masks[:, :, idx] = mask         \n    return masks","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from torch.utils.data import TensorDataset, DataLoader,Dataset\n\nclass CloudDataset(Dataset):\n    def __init__(\n        self,\n        img_ids = None,\n        transforms = None,\n        df = None,\n        mask_dir = None,\n        image_dir = None,\n        npy = False, #Images in numpy format or not\n        ram = False\n    ):\n        self.img_ids = img_ids\n        self.transforms = transforms\n        self.df = df\n        self.mask_dir = mask_dir\n        self.image_dir = image_dir\n        self.npy = npy\n        self.ram = ram\n        if self.ram:\n            self.samples = self.load_samples()\n            \n    def load_samples(self):\n        samples = []\n        print(\"Loading images...\")\n        for image_name in tqdm(self.img_ids):\n            mask = make_mask(image_name, self.mask_dir, self.df)\n            img = get_img(image_name, self.image_dir, self.npy)\n            augmented = self.transforms(image=img, mask=mask)\n            samples.append(augmented)\n        return samples    \n            \n    \n    def __getitem__(self, idx):\n        if self.ram:\n            sample = self.samples[idx]\n            return sample[\"image\"], sample[\"mask\"]\n        \n        image_name = self.img_ids[idx]\n        \n        mask = make_mask(image_name, self.mask_dir, self.df)\n        img = get_img(image_name, self.image_dir, self.npy)\n        \n        augmented = self.transforms(image=img, mask=mask)\n        img = augmented[\"image\"]\n        mask = augmented[\"mask\"]\n        return img, mask\n\n    def __len__(self):\n        return len(self.img_ids)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"imgs_ids = os.listdir(dir_ip)[:N]\n\ntfms1 = albu.Compose([albu.Resize(320, 640),\n                      albu.HorizontalFlip(p=0.5),\n                    ])\ntfms2 = albu.Compose([albu.HorizontalFlip(p=0.5)])\n\nnum_workers = 0\nbs = 16\ndf = train\n\n#Loader1: Load large images then resize + Create masks\ndataset1 = CloudDataset(imgs_ids, tfms1, df, \n                             False, dir_ip, npy=False)\nloader1 = DataLoader(dataset1, batch_size=bs, shuffle=False, num_workers=num_workers)\n\n# Loader2: Load resized images + Create masks\ndataset2 = CloudDataset(imgs_ids, tfms2, df, \n                             False, dir_op, npy=False)\nloader2 = DataLoader(dataset2, batch_size=bs, shuffle=False, num_workers=num_workers)\n\n# Loader3: Load resized images + Load masks\ndataset3 = CloudDataset(imgs_ids, tfms2, False, \n                             dir_op_mask, dir_op, npy=False)\nloader3 = DataLoader(dataset3, batch_size=bs, shuffle=False, num_workers=num_workers)\n\n# Loader4: Load resized images in numpy + Load masks\ndataset4 = CloudDataset(imgs_ids, tfms2, False, \n                             dir_op_mask, dir_op_npy, npy=True)\nloader4 = DataLoader(dataset4, batch_size=bs, shuffle=False, num_workers=num_workers)\n\n# Loader5: Ram loader\ndataset5 = CloudDataset(imgs_ids, tfms2, False, \n                             dir_op_mask, dir_op_npy, npy=True, ram=True)\nloader5 = DataLoader(dataset5, batch_size=bs, shuffle=False, num_workers=num_workers)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Visualize loaders"},{"metadata":{"trusted":true},"cell_type":"code","source":"loaders = [loader1, loader2, loader3, loader4, loader5]\n\nfor i, loader in enumerate(loaders):\n    print(f\"Testing loader{i+1}\")\n    for batch in loader:\n        images, masks = batch\n        visualize(images[0], masks[0])\n        break        ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"# Test runtimes"},{"metadata":{"trusted":true},"cell_type":"code","source":"import time\n\ndef test_bench(loader, epochs=5):\n    runtimes = []\n    for epoch in tqdm(range(epochs)):\n        start = time.time()\n        for batch in loader:\n            images, masks = batch\n        end = time.time()\n        runtimes.append(end - start)\n    return runtimes\n\nruntimes = {}\nfor i, loader in enumerate(loaders):\n    print(f\"Testing loader{i+1}...\")\n    runtimes[f'loader{i+1}'] = test_bench(loader)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"runtimes = pd.DataFrame(runtimes)\nabout = {\n    'loader1': 'Load image and resize | Create mask',\n    'loader2': 'Load resized image | Create mask',\n    'loader3': 'Load resized image.npy | Create mask',\n    'loader4': 'Load resized image.npy | Load mask.npy',\n    'loader5': 'Image RAMLoader | Mask RAMLoader',\n}\nruntimes = runtimes.rename(columns=about)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15,5))\nruntimes.max().plot(kind='barh', title='Max-time (s)')\nplt.show()\nplt.figure(figsize=(15,5))\nruntimes.min().plot(kind='barh', title='Min-time (s)')\nplt.show()\nplt.figure(figsize=(15,5))\nruntimes.mean().plot(kind='barh', title='Average-time (s)')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"runtimes.head(5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Conclusion\nClearly, as expected, RAM loader is the fastest. However, in this competition we can not load all the images and masks in RAM unless we reduce the size a lot. Loading one of them in RAM might work and increase the data loading speed though. "},{"metadata":{"trusted":true},"cell_type":"code","source":"!rm {dir_op} {dir_op_mask} {dir_op_npy} -r","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Thanks for reading!"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}