{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Why shelve?\nReading individual images and resizing them can be slow. Loading the data from disk - especially for very large images like the X-rays here - is one of things that will slow us down in this competition when we train convolutional neural networks. Would it be faster to save the whole data in a pkl type single file that one can access by some index? Yes, but .pkl would not allow parallel access to the data, so you could not have multiple workers reading data.\n\n`shelve` is like pickle, but allows parallel reading and access via dictionary keys - a persistent dictionary, really (see [the documentation](https://docs.python.org/3/library/shelve.html)). \n\n# What does this notebook do?\nIn this notebook, I resized the images to 704 by 704 (you could of course change that so that, say, the shortest dimension is at least 600 pixels or something) and I, of course, only kept one grascale channel. The whole file we end up saving ends up being small enough for our purposes - primarily through saving the image itself as uint8.\n\nI also show what a dataloader can look like. In another competition, I found that [shelve is faster than the alternatives](https://www.kaggle.com/bjoernholzhauer/vinbigdata-chest-x-ray-comparing-dataloader-speed) such as loading pre-sized images - let me know, if I missed obvious alternatives."},{"metadata":{},"cell_type":"markdown","source":"# Implementation using shelve"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\nimport albumentations as A\nimport pickle as pkl\nimport fastcore\nfrom fastcore.parallel import parallel\nimport shelve\nfrom PIL import Image\nfrom tqdm.auto import tqdm\ntrain = pd.read_csv('../input/ranzcr-clip-catheter-line-classification/train.csv')\ntargets = list(train.columns[1:-1])\ntargets","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def read_and_resize(image_id):\n    aug = A.Resize(704,704,always_apply=True)\n    return {'image': aug(image=np.array(Image.open(f'../input/ranzcr-clip-catheter-line-classification/train/{image_id}.jpg')))['image'],\n            'labels': np.array(train.loc[train['StudyInstanceUID']==image_id, targets], dtype=np.int8)}\n\nlist_of_images = list(train['StudyInstanceUID'].values)\n\n# Parallel processing of the image files could be an option, but we run out of memory.\n#tmp_imgs = parallel(read_and_resize, list_of_images, n_workers=4, progress=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for image_id in tqdm(list_of_images):\n    with shelve.open('training_data.db') as myshelf:\n        myshelf.update( { image_id: read_and_resize(image_id)})","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Accessing the data"},{"metadata":{"trusted":true},"cell_type":"code","source":"with shelve.open('training_data.db', flag='r', writeback=False) as myshelf:\n    print(\"------ First training records -------------\")\n    print(train['StudyInstanceUID'].values[0])\n    print( myshelf[train['StudyInstanceUID'].values[0]] )\n    print(\"------ Second training records -------------\")\n    print(train['StudyInstanceUID'].values[1])\n    print( myshelf[train['StudyInstanceUID'].values[1]] )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Bad DataLoader\n\nThis is a logical way of doing it, but it turns out to be pretty slow, because we open the `shelve` file again and again for each item."},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch\nfrom fastai.vision.all import *\n\nclass ClassificationDataset:\n    def __init__(self, StudyInstanceUIDs, augmentations=None): \n        self.StudyInstanceUIDs, self.augmentations = StudyInstanceUIDs, augmentations\n    def __len__(self): return len(self.StudyInstanceUIDs)\n    def __getitem__(self, item):        \n        with shelve.open('training_data.db', flag='r', writeback=False) as myshelf:\n            tmpdict =  myshelf[self.StudyInstanceUIDs[item]]        \n        # May wish to add resizing option with resize=None as default, could look something like this, if these were PIL images:\n        #if self.resize is not None:\n        #    image = image.resize( (self.resize[1], self.resize[0]), resample=Image.BILINEAR)\n        # We add a channel dimension, even though images are just 1 channel, because albumentations wants that\n        image = np.expand_dims(np.array( tmpdict['image'] ), axis=-1)\n        if self.augmentations is not None:\n            image = self.augmentations(image=image)['image']\n        # Go to torch tensor and switch from height-width-channel tto channel-height-width \n        image = torch.tensor( image[:,:,0] )[None, :, :]\n        return image, torch.tensor( tmpdict['labels'], dtype=torch.long) #torch.tensor(image, dtype=torch.float)\n\nimage_size = 512\n    \ntrain_aug = A.Compose([        \n    A.HorizontalFlip(p=0.3),    \n    A.OneOf([ A.Rotate(limit=10, border_mode=4, always_apply=True), \n              A.Rotate(limit=10, border_mode=0, value=[0,0,0], always_apply=True),\n              A.Rotate(limit=10, border_mode=0, value=[128,128,128], always_apply=True)], p=0.75),\n    A.RandomResizedCrop(image_size, image_size, scale=(0.9, 1), always_apply=True),    \n    A.RandomBrightnessContrast(brightness_limit=(-0.05,0.05), contrast_limit=(-0.05, 0.05), p=0.5),\n    A.CLAHE(clip_limit=(0,2), p=0.5),\n    A.GaussNoise(var_limit=[0, 20], p=0.5),\n    A.OneOf([A.JpegCompression(quality_lower=75, quality_upper=99),A.Downscale(scale_min=0.5, scale_max=0.8),], p=0.5),\n    A.IAAPiecewiseAffine(scale=(0, 0.025), p=0.2),    \n    A.CoarseDropout(min_holes=2, max_holes=8, \n                    max_height=int(image_size * 0.15), \n                    max_width=int(image_size * 0.15), \n                    min_height=int(image_size * 0.05), \n                    min_width=int(image_size * 0.05), \n                    fill_value=0, always_apply=False, p=0.5),      \n    A.Normalize(0.4827506 , 0.22004028, max_pixel_value=255.0, always_apply=True)\n])    \n    \n    \ntrain_dataset = ClassificationDataset(StudyInstanceUIDs=list_of_images[0:200], \n                                      augmentations=train_aug)    \n\n# train_dataset.__getitem__(0) # Can do that to test getting one item\n\ntrain_loader = DataLoader(dataset=train_dataset, bs=32, shuffle=True, n_workers=2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"next(iter(train_loader))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Good DataLoader\n\nThere's many approaches we could take to DataLoaders, the previous naive idea turns out to be too slow. For example, it takes almost half a minute for a batch of 32!\n\nThe version below that loads a whole batch in one go (and importantly only opens the `shelve` file only once per batch) is massively faster."},{"metadata":{"trusted":true},"cell_type":"code","source":"def getbatch(items, augmentations=None):\n    with shelve.open('training_data.db', flag='r', writeback=False) as myshelf:\n        tmpdict =  { key: myshelf[key] for key in items }\n        \n    # May wish to add resizing option with resize=None as default, could look something like this, if these were PIL images:\n    #if self.resize is not None:\n    #    image = image.resize( (self.resize[1], self.resize[0]), resample=Image.BILINEAR)\n            \n    # We add a channel dimension, even though images are just 1 channel, because albumentations wants that        \n    images = [ np.expand_dims(tmpdict[key]['image'], axis=-1) for key in items ]    \n    # If augmentations are requested, do these one image at a time\n    if augmentations is not None:\n        images = [ augmentations(image=image)['image'] for image in images]\n    # We turn the list of image arrays into a 4-D numpy tensor.\n    # Go to torch tensor and switch from item-height-width-channel tto channel-height-width     \n    images = torch.tensor( np.array(images)[:,:,:,0] )[:,None, :, :]    \n    labels = torch.tensor( [ tmpdict[key]['labels'][0] for key in items ], dtype=torch.long)\n        \n    return images, labels\n\n# # You can test this works as expected via:\n# x,y = getbatch(items=list_of_images[:32], augmentations=train_aug)\n# x.shape # torch.Size([32, 1, 512, 512])\n# y.shape # torch.Size([32, 11])\n\nclass BatchedParallelDataLoader:\n\n    def __init__(self, image_ids, augmentations, batch_size: int=32, shuffle: bool=False, drop_last: bool=False, n_workers: int=1):\n        self.dataset_len, self.batch_size, self.shuffle, self.drop_last, self.n_workers = len(image_ids), batch_size, shuffle, drop_last, n_workers\n        self.image_ids, self.augmentations = np.array(image_ids), augmentations                    \n\n        # Calculate # batches\n        n_batches, remainder = divmod(self.dataset_len, self.batch_size)\n        if (remainder > 0) & (drop_last==False):\n            n_batches += 1\n            self.n_items = self.dataset_len\n        else:\n            self.n_items = self.n_batches * self.batch_size\n        self.n_batches = n_batches\n        self.batch_list = [i for i in range(self.n_items)]        \n\n    def __iter__(self):\n        if self.shuffle:\n            ridx = torch.randperm(self.dataset_len)\n            self.batch_list = [ridx[i] for i in range(self.n_items)]\n            \n        chunks = [ list(self.image_ids[ self.batch_list[i:min(self.n_items, i+self.batch_size)]]) for i in range(0, self.n_items, self.batch_size) ]        \n        with ProcessPoolExecutor(self.n_workers) as ex:\n            yield from ex.map(getbatch, chunks, augmentations=self.augmentations)\n\n    def __len__(self):\n        return self.n_batches\n    \ntrain_dl = BatchedParallelDataLoader(n_workers=2, image_ids=list_of_images, augmentations=train_aug, batch_size=32)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}