{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Optimization tips\nOptimizing your code is usually a good practice, but it is inevitable in a limited environment like Kaggle's Code Competitions.\nIn this notebook, I'd like to show you some tips on how you can optimize your code. Hopefully, using these you won't run out of time or memory.\n\nQuick reminder:\n> Submissions to this competition must be made through Notebooks. In order for the \"Submit to Competition\" button to be active after a commit, the following conditions must be met:\n- CPU Notebook <= 9 hours run-time\n- GPU Notebook <= 2 hours run-time\n- No internet access enabled\n- External data is allowed, and you are encouraged to train your model offline and use your Notebook for inference.\n- Submission file must be named \"submission.csv\"\n>\n>+1\n- Available memory: 16Gb\n\n\n\n## Summary\n- Use script instead of notebook\n- Import only things you need\n- Use logs instead of tqdm\n- Cleanup after usage\n- Load parquet files once\n- Do not load data you don't need\n- Check your dtypes\n- Preprocess your images once\n- Use CUDA for preprocessing\n- Do not use albumentation (inference)\n- Only use 3 channels if you really need it\n- Process in batches\n- Optimized TTA\n\n\nI rate the tips by effect, using medal icons (from 1 to 5)\n- 🥇 minor effect\n- 🥇🥇🥇🥇🥇 significant effect on memory usage/running time\n\n\n## Implementation\nYou can find the implementation (most) of these tips in this kernel:\n\n[https://www.kaggle.com/pestipeti/fast-ensemble-5-folds-20-minutes](https://www.kaggle.com/pestipeti/fast-ensemble-5-folds-20-minutes)"},{"metadata":{},"cell_type":"markdown","source":"## TIP \\#1: 🥇 Use `script` instead of `notebook`.\nIt is not significant, but we have a bit more free memory by using a script (`17.09Gb`) instead of a notebook (`16.08Gb`).\nJupyter has lots of great features, but in a background run, you can not use any of those.\n\n\n\n| Notebook  | Script  |\n|:-:|:-:|\n|![https://albumizr.com/ia/09b6727ee71143d44313b7fd9a23c9ca.jpg](https://albumizr.com/ia/09b6727ee71143d44313b7fd9a23c9ca.jpg) | ![https://albumizr.com/ia/ea1a157604506f4afc9ebe301871a68d.jpg](https://albumizr.com/ia/ea1a157604506f4afc9ebe301871a68d.jpg) |\n|  | * |\n\n\nIf you insist on using Jupyter Notebook, then make sure you reset the namespace ([doc](https://ipython.readthedocs.io/en/stable/interactive/magics.html#magic-reset)) regularly."},{"metadata":{},"cell_type":"markdown","source":"## TIP \\#2: 🥇 Import what you need only\nIn public kernels, there are lots of unnecessary imports. It may not use too many resources, but why would we import for example `Plotly` in an inference kernel.\n"},{"metadata":{},"cell_type":"markdown","source":"## TIP \\#3: 🥇 Use logs instead of tqdm\nSame as *Tip \\#2*, using TQDM in an inference kernel is pointless."},{"metadata":{},"cell_type":"markdown","source":"## TIP \\#4: 🥇 Cleanup after usage\nRemove every variable once you don't need it anymore."},{"metadata":{"trusted":true},"cell_type":"code","source":"from sys import getsizeof, getrefcount\n\ni = 1\nch = 'c'\nst = \"asasdf\"\nint_list = [x for x in range(10)]\nstr_list = [str(x) for x in range(10)]\n\nprint(\"Size of an int: {} bytes\".format(getsizeof(i)))\nprint(\"Size of a char: {} bytes\".format(getsizeof(ch)))\nprint(\"Size of a string: {} bytes\".format(getsizeof(st)))\nprint(\"Size of list of ints: {} bytes\".format(getsizeof(int_list)))\nprint(\"Size of list of strings: {} bytes\".format(getsizeof(str_list)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(int_list)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del i\ndel ch\ndel st\ndel int_list, str_list","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Make sure that all of the references deleted.\nprint(\"Number of references to variable `int_list`: {}\".format(getrefcount(int_list)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## TIP \\#5: 🥇🥇🥇🥇 Load parquet files once"},{"metadata":{"trusted":true},"cell_type":"code","source":"import pyarrow.parquet as pq","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%timeit\nparq = pq.read_pandas('/kaggle/input/bengaliai-cv19/train_image_data_0.parquet').to_pandas()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This code is more or less okay if you are using one model (one fold) only.\n```\nfor i in range(4):\n    parq = pq.read_pandas('/kaggle/input/bengaliai-cv19/train_image_data_0.parquet').to_pandas()\n    # pre-process parquet file\n    # ...\n    # predict\n    # ...\n```\n\nBut if you want to ensemble many folds/models:\n```\nfor model_idx, model in enumerate(models):\n    for i in range(4):\n        parq = pq.read_pandas('/kaggle/input/bengaliai-cv19/train_image_data_0.parquet').to_pandas()\n        # pre-process parquet file\n        # ...\n        # predict\n        # ...\n```\nThis one is much worse. In this case, I loaded (and preprocessed) the same parquet files multiple times.\nBased on the `timeit` above, it takes ~1 minutes 30 seconds to load one parquet file. \nIf you have a similar code (like my bad one above) than for a 5-folds ensemble, your script wastes ~30 minutes to loading the data. You can reduce this time to ~5-6 minutes if you load the data once and keep it in memory."},{"metadata":{},"cell_type":"markdown","source":"## TIP \\#6: 🥇🥇 Do not load data you don't need\nUse `pyarrow.parquet.read_pandas`'s `columns` argument to exclude the `image_id`. It is a text field (takes lots of memory), and it is generated from the index: `Test_` + `index`. You can predict/submit without this column.\n\n```\nimport pyarrow.parquet as pq\n\n# This is a pandas DataFrame\nparq_df = pq.read_pandas('... parquet file...', column=[str(x) for x in range(32332)]).to_pandas()\n```\n"},{"metadata":{},"cell_type":"markdown","source":"## TIP \\#7: 🥇🥇🥇 Check your dtypes\nThe fastest way for running your code is if you can keep all the data (test images) in memory. We have lots of samples, so we have to use a few tricks.\n\nFor storing images (pixel values), we only need `uint8` data type. The size of an `uint8` variable is 1 byte.  Most of the preprocessing step can be calculated using `uint8`. If you make complicated steps, don't forget to convert the images back to `uint8`.\n\nIn the train set we have 200840 samples (test set has similar), if we store all of the images in memory, it would take `200840 * 137 * 236` bytes (~6.05Gb)"},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\n\nimages = np.random.randint(low=0, high=255, size=(200840, 137 * 236), dtype=np.uint8)\nprint(\"{0:.2f}Gb\".format(images.nbytes / (1024*1024*1024)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"float_images = 255.0 * np.random.rand(1, 137 * 236)\nprint(\"Number of bytes per (float64) image: {}\".format(float_images.nbytes))\nprint(\"Number of bytes per (float32) image: {}\".format(float_images.astype(np.float32).nbytes))\n\nprint(\"Memory usage of the full dataset (float32): {0:.2f}Gb\".format(float_images.astype(np.float32).nbytes * 200840 / (1024*1024*1024)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## TIP \\#8: 🥇🥇🥇🥇 Preprocess your images once\nIf you are ensembling multiple models/folds, it is crucial to preprocess the samples before you start predicting. You don't want to do the same cutting/padding/resizing steps many times.\n\n**Note**: You can not do any calculation during preprocessing if the result would be `float`. The entire dataset would not fit in memory. One of these steps, for example, is normalization."},{"metadata":{},"cell_type":"markdown","source":"## TIP \\#9: 🥇🥇 Use CUDA for preprocessing\nYou can save lots of time if you do your preprocessing on the GPU.\n- Load all of the samples to memory (`uint8`)\n- Move everything to CUDA\n- Preprocess your images one-by-one (or in batches if possible)\n- Move everything back to RAM"},{"metadata":{},"cell_type":"markdown","source":"## TIP \\#10: 🥇🥇 Do not use albumentation (inference)\nAt inference time after preprocessing the images, usually only two things left: normalization and convert to CUDA tensor. Using Albumentation (or `torch.transform`) typically this is done in the `DataLoader`. The problem with this is that the sampler requests the images one-by-one for building up the next batch. It is faster if you do the final transformations in batches.\n\nSo, instead of this:\n```\nclass MyDataLoader():\n\n    ...\n\n    def __getitem__(self, idx):\n        ...\n        image = self.transform(image) # albumentation or torch transforms\n        return image\n\n    ...\n```\n\nYou should do this (in your prediction loop):\n```\n...\nfor batch_idx, images in enumerate(test_loader):\n    \n    # You can do the normalization step in your\n    # model's forward method\n    \n    # final transforms / TTA should be here\n\n    ...\n\n    images = images.float().cuda()\n    \n    outputs = model(images)\n    \n    ...\n    \n...\n```"},{"metadata":{},"cell_type":"markdown","source":"## TIP \\#11: 🥇🥇 Only use 3 channels if you really need it\n3 channels triple the memory footprint of the samples (probably won't fit)\n\nMost of the models take images with 3 channels as input, but you can easily modify that.\n\n```\nclass BengaliModel(nn.Module):\n\n    def __init__():\n        \n        self.backbone = torchvision.models.resnet34(pretrained=True)\n   \n        old_conv1 = self.backbone.conv1\n        self.backbone.conv1 = nn.Conv2d(1, 64, kernel_size=7,\n            stride=2, padding=3, bias=False)\n        \n        # Here I copied only the first channel's weights, but you can use\n        # average of the 3 channels as well.\n        with torch.no_grad():\n            self.backbone.conv1.weight = nn.Parameter(\n                old_conv1.weight.data[:, 0, :, :].unsqueeze(1))\n\n```\n"},{"metadata":{},"cell_type":"markdown","source":"## TIP \\#12: 🥇🥇🥇 Process in batches\n- Always use batches, if possible. For preprocessing, predicting, etc.\n- Try to eliminate all for loops.\n- In your prediction loop use the largest batch size that fits in memory\n- If you ensemble different size of models use different size of batches too (You only have to re-create the dataloader because of the different batch size)\n"},{"metadata":{},"cell_type":"markdown","source":"## TIP \\#13: 🥇🥇 Optimized TTA\nThere are lots of \"easy\" samples, where all of your models (ResNet-18 as well) predict confidently. In these cases, it is pointless to average two (or more) 0.98 predictions. You can set a threshold (using the validation set) and generate TTA predictions only in uncertain cases.\n"},{"metadata":{},"cell_type":"markdown","source":"--------------------------"},{"metadata":{},"cell_type":"markdown","source":"**Thanks for reading.** Please vote if you find these tips useful."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}