{"cells":[{"metadata":{},"cell_type":"markdown","source":"This kernel shows how training time can be considerably improved. The approach is demonstrated using competition data from the [APTOS 2019 Blindness Detection challenge](https://www.kaggle.com/c/aptos2019-blindness-detection), but we can easily upscale it to also include old competition data from 2015 in training (around ~40.000 images in total)."},{"metadata":{},"cell_type":"markdown","source":"## Objective"},{"metadata":{},"cell_type":"markdown","source":"Most public kernels currently perform image pre-processing dynamically during training. This is memory-efficient, because only data needed for the current batch is loaded on memory. However, it also increases training time, because pre-processing is repeated for each image in each epoch. For example, for a 5-fold CV with 10 epochs each, we would apply the same pre-processing 50 times to each image. My objective was to move image pre-processing out of the training loop, and therefore to apply it only once during training.\n\nNote: I define image 'pre-processing' in this kernel as any processing steps that are uniformly applied to each image, such as resizing the image, cropping, or applying Ben's pre-processing. In contrast, data augmentation, such as random flipping or random rotation, is randomly applied to each image in each epoch and therefore has to stay inside the training loop.\n"},{"metadata":{},"cell_type":"markdown","source":"## Method"},{"metadata":{},"cell_type":"markdown","source":"First, what didn't work:\n\n* Loading all pre-processed images into memory (as demonstrated in [this excellent kernel](https://www.kaggle.com/xhlulu/aptos-2019-densenet-keras-starter)). This works very well if we only use data from this year’s competition, but I had memory issues when training old competition data (~40.000 images are too much for the kernel memory). \n* Saving all pre-processed images as .png files and loading them during training. This led to a kernel error saying that too many output files were produced. \n\nWhat did work in the end: Saving all pre-processed images in a single HDF5-file (see below).\n"},{"metadata":{},"cell_type":"markdown","source":"## Speed improvement "},{"metadata":{},"cell_type":"markdown","source":"To benchmark the speed improvement of the approach described in this kernel I used these references:\n\n* Pre-processing is entirely based on [Neuron Engineers kernel](https://www.kaggle.com/ratthachat/aptos-updated-preprocessing-ben-s-cropping).\n* Training is entirely based on [Chanhus EfficientNet training kernel](https://www.kaggle.com/chanhu/eye-efficientnet-pytorch-lb-0-777), which is currently the best-scoring public kernel. That kernel is used as the reference kernel in the table below.\n\nSpeed improvements were substantial: \n\n\n| | Duration pre-processing | Duration 1 epoch | Total training duration (10 epochs) |\n| --- | --- | --- | --- |\n| Reference kernel | -- | ~600s | ~6.000s |\n| This kernel | ~500s | ~35s | ~850s |\n\n<br>\n\nIn another (private) kernel, I trained a model with the same EfficientNet b0 architecture using old and new data in around 400s per epoch (pre-processing took around 5000s). \n  \nSpeed improvements are biggest when using k-fold CV. A 5-fold CV of the reference kernel would take around 30.000s. With the approach from this kernel, this can be reduced to around 2.500s. "},{"metadata":{},"cell_type":"markdown","source":"## Demonstration"},{"metadata":{},"cell_type":"markdown","source":"The code in this kernel is almost entirely from the reference kernel (https://www.kaggle.com/chanhu/eye-efficientnet-pytorch-lb-0-777). All modifications made by me are commented."},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"! pip install -v --no-cache-dir --global-option=\"--cpp_ext\" --global-option=\"--cuda_ext\" ../input/nvidiaapex/repository/NVIDIA-apex-39e153a","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true,"scrolled":true,"_kg_hide-input":false,"_kg_hide-output":false},"cell_type":"code","source":"import cv2\nimport matplotlib.pyplot as plt\nfrom os.path import isfile\nimport torch.nn.init as init\nimport torch\nimport torch.nn as nn\nimport numpy as np\nimport pandas as pd \nimport os\nfrom PIL import Image, ImageFilter\nprint(os.listdir(\"../input\"))\nfrom sklearn.model_selection import train_test_split, StratifiedKFold\nfrom torch.utils.data import Dataset\nfrom torchvision import transforms\nfrom torch.optim import Adam, SGD, RMSprop\nimport time\nfrom torch.autograd import Variable\nimport torch.functional as F\nfrom tqdm import tqdm\nfrom sklearn import metrics\nimport urllib\nimport pickle\nimport cv2\nimport torch.nn.functional as F\nfrom torchvision import models\nimport seaborn as sns\nimport random\nfrom apex import amp\nimport sys","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We'll use [h5py](https://www.h5py.org/) as the HDF5-interface and multiprocessing for image pre-processing."},{"metadata":{"trusted":true},"cell_type":"code","source":"import h5py\nfrom multiprocessing import Pool\nfrom itertools import repeat\nfrom timeit import default_timer as timer","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"package_path = '../input/efficientnet/efficientnet-pytorch/EfficientNet-PyTorch/'\nsys.path.append(package_path)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from efficientnet_pytorch import EfficientNet","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def seed_everything(seed):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.backends.cudnn.deterministic = True","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"num_classes = 1\nseed_everything(1234)\nlr          = 1e-3\nIMG_SIZE    = 256","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"766f44c87272f67d632e519dce11cf54a3382696"},"cell_type":"code","source":"train      = '../input/aptos2019-blindness-detection/train_images/'\ntest       = '../input/aptos2019-blindness-detection/test_images/'\ntrain_csv  = pd.read_csv('../input/aptos2019-blindness-detection/train.csv')\n\ntrain_df, val_df = train_test_split(train_csv, test_size=0.1, random_state=2018, stratify=train_csv.diagnosis)\ntrain_df.reset_index(drop=True, inplace=True)\nval_df.reset_index(drop=True, inplace=True)\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#The Code from: https://www.kaggle.com/ratthachat/aptos-updated-albumentation-meets-grad-cam\n\ndef crop_image1(img,tol=7):\n    # img is image data\n    # tol  is tolerance\n        \n    mask = img>tol\n    return img[np.ix_(mask.any(1),mask.any(0))]\n\ndef crop_image_from_gray(img,tol=7):\n    if img.ndim ==2:\n        mask = img>tol\n        return img[np.ix_(mask.any(1),mask.any(0))]\n    elif img.ndim==3:\n        gray_img = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)\n        mask = gray_img>tol\n        \n        check_shape = img[:,:,0][np.ix_(mask.any(1),mask.any(0))].shape[0]\n        if (check_shape == 0): # image is too dark so that we crop out everything,\n            return img # return original image\n        else:\n            img1=img[:,:,0][np.ix_(mask.any(1),mask.any(0))]\n            img2=img[:,:,1][np.ix_(mask.any(1),mask.any(0))]\n            img3=img[:,:,2][np.ix_(mask.any(1),mask.any(0))]\n    #         print(img1.shape,img2.shape,img3.shape)\n            img = np.stack([img1,img2,img3],axis=-1)\n    #         print(img.shape)\n        return img","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"n_cores = 2\nh5_file_name = 'train_images.h5' # name of the hdf5 file used throughout the notebook","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Pre-processing from https://www.kaggle.com/ratthachat/aptos-updated-preprocessing-ben-s-cropping :"},{"metadata":{"trusted":true},"cell_type":"code","source":"def crop_and_resize_images(image_name, image_folder, target_size):\n    image_path = os.path.join(image_folder, image_name + '.png')\n\n    image = cv2.imread(image_path)\n    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n    image = crop_image_from_gray(image)\n    image = cv2.resize(image, (target_size, target_size))\n    image=cv2.addWeighted(image, 4, cv2.GaussianBlur(image, (0,0), 10), -4, 128)\n\n    return image","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In the following function, we \n\n1. create an empty hdf5 file,\n2. pre-process images using multiprocessing\n3. save pre-processed images in the hdf5 file.\n\nComments to 1:\n* We create an empty dataset inside the hdf5 file with compression='gzip'. Compression can be removed if we only want to train images from the current competition. However, it is necessary if we want to train images from 2015; without compression, the resulting hdf5 file might become too big for the Kaggle kernel and throw an error. \n* Choosing an adequate chunk size for the dataset and access pattern is crucial. During testing, I used h5py's 'auto'-option to select a chunk size. As a result, training time of one epoch was around 25m (instead of 35s in the current version of this kernel). As our Dataloader only ever accesses one image at a time, we use a chunk size that corresponds to one image. This means that each image is compressed separately inside the hdf5 file.\n\nComments to 2:\n* We pre-process images in batches of 1000 to prevent memory-issues if we train images from 2015. If we only want to train images from the current competition, we could train all images in a single batch.\n\nComments to 3:\n* We add a dataset with image names to the hdf5 file. This is later used as a lookup table when retrieving images from the hdf5 file.\n\n\n\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_hdf5_file(image_names, image_folder, target_size, h5_file_name):\n   \n    # create hdf5 file and datasets\n    hdf5_file = h5py.File(h5_file_name, 'w', libver='latest')\n    \n    # compression 'gzip' and adequate chunk-size seems to be the best compromise between file size and access speed\n    hdf5_file.create_dataset('images', (0, target_size, target_size, 3), maxshape=(None, target_size, target_size, 3), compression='gzip',\n                             compression_opts=2, chunks=(1, target_size, target_size, 3), dtype=np.uint8)\n    hdf5_file.close()\n\n    # create batches of images to prevent holding all images into memory at the same time\n    batch_size = 1000\n    n_batches = np.ceil(len(image_names) / batch_size).astype(np.int)\n\n    image_i = 0\n    for i, batch in enumerate(range(n_batches)):\n        \n        start_time = timer()\n\n        batch_start_image_idx = image_i\n        batch_stop_image_idx = image_i + batch_size\n\n        image_i = batch_stop_image_idx\n\n        if batch_stop_image_idx > len(image_names):\n            batch_stop_image_idx = len(image_names)\n            \n        print(f'batch {i+1}/{n_batches}, precessing images {batch_start_image_idx} to {batch_stop_image_idx - 1}')\n\n        image_names_in_batch = image_names.iloc[batch_start_image_idx:batch_stop_image_idx]\n\n        with Pool(n_cores) as pool:\n            images_in_batch = pool.starmap(crop_and_resize_images, \n                                           zip(image_names_in_batch, \n                                               repeat(image_folder), \n                                               repeat(target_size)))\n        \n        images_in_batch = np.asarray(images_in_batch)\n\n        time_elapsed = timer() - start_time\n        print(f'batch processing finished in {(time_elapsed // 60):.0f}m {(time_elapsed % 60):.0f}s, inserting data into h5 file')\n\n        with h5py.File(h5_file_name, 'a') as hdf5_file:\n            # dynamically increase the size of the dataset:\n            hdf5_file['images'].resize((hdf5_file['images'].shape[0] + images_in_batch.shape[0]), axis=0)\n            \n            hdf5_file['images'][-images_in_batch.shape[0]:] = images_in_batch\n\n    # add dataset with image names to hdf5 file \n    with h5py.File(h5_file_name, 'a') as hdf5_file:\n        assert hdf5_file['images'].shape[0] == len(image_names)\n        hdf5_file.create_dataset('id_code', data=image_names.values.astype('S'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ncreate_hdf5_file(image_names=train_csv['id_code'], image_folder=train, target_size=IMG_SIZE, h5_file_name=h5_file_name)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f'Size of h5 file: {(os.path.getsize(h5_file_name) / float(1 << 20)):.0f} mb')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"The following function returns a single image from the hdf5 file:"},{"metadata":{"trusted":true},"cell_type":"code","source":"def open_image_from_hdf5_file(image_name):\n    with h5py.File(h5_file_name, 'r') as hdf5_file:\n        image_names_in_h5 = np.array(hdf5_file.get('id_code'))\n        image_idx_in_h5 = np.where(image_names_in_h5 == image_name.encode())[0][0]\n        image = hdf5_file['images'][image_idx_in_h5]\n        image = transforms.ToPILImage()(image)\n        return image","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"64d7e44b053ac654c681e77b04de74ba32020fbd"},"cell_type":"code","source":"def p_show(imgs, label_name=None, per_row=3):\n    n = len(imgs)\n    rows = (n + per_row - 1)//per_row\n    cols = min(per_row, n)\n    fig, axes = plt.subplots(rows,cols, figsize=(15,15))\n    for ax in axes.flatten(): ax.axis('off')\n    for i,(p, ax) in enumerate(zip(imgs, axes.flatten())): \n        img = open_image_from_hdf5_file(p)\n        ax.imshow(img)\n        ax.set_title(train_df[train_df.id_code == p].diagnosis.values)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b4739904397dd22d058c36769034b91964dcb9fe"},"cell_type":"code","source":"imgs = []\nfor p in train_df.id_code:\n    imgs.append(p)\n    if len(imgs) == 16: break\np_show(imgs)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The following dataset is modified to work with our hdf5 file. One thing to note is that the image dataset in the hdf5 is opened only once in the __getitem__ method. This seems to considerably improve access speed. More information can be found here: https://discuss.pytorch.org/t/dataloader-when-num-worker-0-there-is-bug/25643/16"},{"metadata":{"trusted":true,"_uuid":"21908baa8df4e398b0d49a5146ce544504637c5a"},"cell_type":"code","source":"class MyDatasetHDF5(Dataset):\n    \n    def __init__(self, dataframe, h5_file_name, transform=None):\n        self.df = dataframe\n        self.transform = transform\n        self.h5_file_name = h5_file_name\n        \n        with h5py.File(h5_file_name, 'r') as hdf5_file:\n            self.image_names_in_h5 = np.array(hdf5_file.get('id_code')) \n            \n        self.h5_dataset = None\n    \n    def __len__(self):\n        return len(self.df)\n    \n    def __getitem__(self, idx):\n        \n        label = self.df.diagnosis.values[idx]\n        label = np.expand_dims(label, -1)\n        \n        image_name = self.df.id_code.values[idx]\n        \n        # here, we get the index of the image in the hdf5 file:\n        image_idx_in_h5 = np.where(self.image_names_in_h5 == image_name.encode())[0][0] \n        \n        if self.h5_dataset is None:\n            self.h5_dataset = h5py.File(self.h5_file_name, 'r')['images']\n        \n        image = self.h5_dataset[image_idx_in_h5]\n        image = transforms.ToPILImage()(image)\n        \n        if self.transform:\n            image = self.transform(image)\n        \n        return image, label","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f590638fd07b9aefe2210a39612ac77e0689c0c1"},"cell_type":"code","source":"train_transform = transforms.Compose([\n    transforms.RandomHorizontalFlip(),\n    transforms.RandomRotation((-120, 120)),\n    transforms.ToTensor(),\n    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])])\n\ntrainset     = MyDatasetHDF5(train_df, h5_file_name=h5_file_name, transform =train_transform)\ntrain_loader = torch.utils.data.DataLoader(trainset, batch_size=32, shuffle=True, num_workers=0)\nvalset       = MyDatasetHDF5(val_df, h5_file_name=h5_file_name, transform   =train_transform)\nval_loader   = torch.utils.data.DataLoader(valset, batch_size=32, shuffle=False, num_workers=0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Note: Setting the number of workers in the Dataloader to anything else than 0 results in an error. \n\nThe rest of the code is left exactly as it is in the reference kernel. "},{"metadata":{"trusted":true,"_uuid":"5600b405f51d623922c315eef30612e91205bfff","_kg_hide-output":true},"cell_type":"code","source":"model = EfficientNet.from_name('efficientnet-b0')\nmodel.load_state_dict(torch.load('../input/efficientnet-pytorch/efficientnet-b0-08094119.pth'))\nin_features = model._fc.in_features\nmodel._fc = nn.Linear(in_features, num_classes)\nmodel.cuda()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"optimizer = torch.optim.Adam(model.parameters(), lr=lr, weight_decay=1e-5)\ncriterion = nn.MSELoss()\nscheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=5, gamma=0.1)\nmodel, optimizer = amp.initialize(model, optimizer, opt_level=\"O1\",verbosity=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c338feda0eee741964b4c3d736c30b1e0a7e3ace"},"cell_type":"code","source":"def train_model(epoch):\n    model.train() \n        \n    avg_loss = 0.\n    optimizer.zero_grad()\n    for idx, (imgs, labels) in enumerate(train_loader):\n        imgs_train, labels_train = imgs.cuda(), labels.float().cuda()\n        output_train = model(imgs_train)\n        loss = criterion(output_train,labels_train)\n        with amp.scale_loss(loss, optimizer) as scaled_loss:\n            scaled_loss.backward()\n        optimizer.step() \n        optimizer.zero_grad() \n        avg_loss += loss.item() / len(train_loader)\n        \n    return avg_loss\n\ndef test_model():\n    \n    avg_val_loss = 0.\n    model.eval()\n    with torch.no_grad():\n        for idx, (imgs, labels) in enumerate(val_loader):\n            imgs_vaild, labels_vaild = imgs.cuda(), labels.float().cuda()\n            output_test = model(imgs_vaild)\n            avg_val_loss += criterion(output_test, labels_vaild).item() / len(val_loader)\n        \n    return avg_val_loss","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3562bd2ec1b0650519ca196bfc0e60eb139ca180"},"cell_type":"code","source":"best_avg_loss = 100.0\nn_epochs      = 10\n\nfor epoch in range(n_epochs):\n    \n    print('lr:', scheduler.get_lr()[0]) \n    start_time   = time.time()\n    avg_loss     = train_model(epoch)\n    avg_val_loss = test_model()\n    elapsed_time = time.time() - start_time \n    print('Epoch {}/{} \\t loss={:.4f} \\t val_loss={:.4f} \\t time={:.2f}s'.format(\n        epoch + 1, n_epochs, avg_loss, avg_val_loss, elapsed_time))\n    \n    if avg_val_loss < best_avg_loss:\n        best_avg_loss = avg_val_loss\n        torch.save(model.state_dict(), 'weight_best.pt')\n    \n    scheduler.step()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"## Further improvements and ideas"},{"metadata":{},"cell_type":"markdown","source":"* The resulting hdf5 file can be used as a input dataset in subsequent kernels. This leads to a further improvement in training speed, because no pre-preprocessing has to be performed during training.\n* Pre-processing could be moved to a dedicated CPU kernel, where we could use 4 CPUs for pre-processing (instead of the 2 available CPUs in the GPU kernels).\n* Using multiple workers in the Dataloader could further improve training speed. According to [this discussion](https://discuss.pytorch.org/t/dataloader-when-num-worker-0-there-is-bug/25643/16), it should be possible in theory to do so. \n* For inference, the approach from this kernel is not relevant if we submit only one model. However, for ensembling, this approach could lead to a large speed boost if multiple models are based on identical pre-processing. \n"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}