{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Introduction\nMotivation - We go through many kernels and repos and try to understand the what the model is doing. And as you know, apart from the fundemental differences in the architecture and training procedure we see lots of variation in the coding style like the way the training loop is, what happens at the end of each training or validation epoch. So you have to spend some time to understand things that does not really contribute to the fundemental approach. And even for the person developing the model, he spends a good time to write boilerplate code.  \n\nThis kernel is a quick introduction to [PyTorch Lightning](https://github.com/PyTorchLightning/pytorch-lightning#how-do-i-do-use-it) a PyTorch wrapper for ML researchers. This automates non essential procedures and enforces a good coding style to make machine learning solutions much more consistent and reproducible. \"More of a style guide than a framework\".  \n\n![lightning_logo.svg](attachment:lightning_logo.svg)\n\ntl;dr just jump into [The Lightning Module](#The-Lightning-Module) section which has the stuff I wanted to share!","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"attachments":{"lightning_logo.svg":{"image/svg+xml":"PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0idXRmLTgiIHN0YW5kYWxvbmU9Im5vIj8+CjxzdmcgeG1sbnM6ZGM9Imh0dHA6Ly9wdXJsLm9yZy9kYy9lbGVtZW50cy8xLjEvIiB4bWxuczpjYz0iaHR0cDovL2NyZWF0aXZlY29tbW9ucy5vcmcvbnMjIiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiIHhtbG5zOnN2Zz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyIgeG1sbnM6c29kaXBvZGk9Imh0dHA6Ly9zb2RpcG9kaS5zb3VyY2Vmb3JnZS5uZXQvRFREL3NvZGlwb2RpLTAuZHRkIiB4bWxuczppbmtzY2FwZT0iaHR0cDovL3d3dy5pbmtzY2FwZS5vcmcvbmFtZXNwYWNlcy9pbmtzY2FwZSIgaWQ9InN2ZyIgdmVyc2lvbj0iMS4xIiB3aWR0aD0iNDcuOTk5OTg1IiBoZWlnaHQ9IjQ3Ljk5OTk0MyIgdmlld0JveD0iMCAwIDQ3Ljk5OTk4NSA0Ny45OTk5NDMiIGlua3NjYXBlOnZlcnNpb249IjAuOTIuMyAoMjQwNTU0NiwgMjAxOC0wMy0xMSkiPgogIDxtZXRhZGF0YSBpZD0ibWV0YWRhdGExMyI+CiAgICAKICA8L21ldGFkYXRhPgogIDxkZWZzIGlkPSJkZWZzMTEiLz4KICAKICA8cGF0aCBzdHlsZT0iZmlsbDojZmJmYmZiO2ZpbGwtcnVsZTpldmVub2RkO3N0cm9rZTpub25lO3N0cm9rZS13aWR0aDowLjEyMDA4MzkxIiBkPSJtIDI2Ljk2MTI5NCw1LjE3MDQ1MTkgYyAtMC4xNjc2NCwwLjEwMjY3IC0xMi4zNjU2NCwxMi4yODk2MzAxIC0xMi41NTM5MywxMi41NDI3NzAxIC0wLjA2OTUsMC4wOTMzIC0wLjEzNzYyLDAuMTgwNDggLTAuMTUxMzEsMC4xOTM2OSAtMC4wMjYyLDAuMDI1MiAtMC4yMjQzMiwwLjQzNTE5IC0wLjI5MDM2LDAuNjAwNDIgLTAuMTc1NDQsMC40Mzk3NSAtMC4xNzQxMiwxLjMzMTYxIDAuMDAzLDEuNzc3MjQgMC4yNTI3OCwwLjYzNzI5IDAuMjY0NzksMC42NTAyNiA0LjczMDU5LDUuMTE4ODIgMi4zMzgzOSwyLjMzOTg0IDQuMjUxNTcsNC4yODA2MyA0LjI1MTU3LDQuMzEyODEgMCwwLjA2OTYgLTAuNzU5NTMsMi4zOTU0NCAtMC44MjE2MiwyLjUxNjEyIC0wLjAyMzcsMC4wNDU5IC0wLjA0MywwLjEyMjYxIC0wLjA0MywwLjE3MDUyIDAsMC4wNDc5IC0wLjAxNzcsMC4xMDc2IC0wLjAzOTQsMC4xMzI2OSAtMC4wMjE2LDAuMDI1MSAtMC4xMTAzNSwwLjI3MjU5IC0wLjE5NzA1LDAuNTQ5OTkgLTAuMDg2OCwwLjI3NzM5IC0wLjE3NzEzLDAuNTQxOTQgLTAuMjAwNzgsMC41ODc4MSAtMC4wMjM4LDAuMDQ1OSAtMC4wNDMxLDAuMTIyNiAtMC4wNDMxLDAuMTcwNTIgMCwwLjA0NzkgLTAuMDE3NywwLjEwNzU5IC0wLjAzOTQsMC4xMzI2OSAtMC4wMjE2LDAuMDI1MSAtMC4xMTAzNiwwLjI3MjU5IC0wLjE5NzA2LDAuNTQ5OTkgLTAuMDg2OCwwLjI3NzM5IC0wLjE3NzEyLDAuNTQxOTMgLTAuMjAwNzgsMC41ODc4MSAtMC4wMjM4LDAuMDQ1OSAtMC4wNDMxLDAuMTI1NiAtMC4wNDMxLDAuMTc3MjQgMCwwLjA1MTYgLTAuMDIxNiwwLjEwNzIzIC0wLjA0OCwwLjEyMzU3IC0wLjAyNjQsMC4wMTYzIC0wLjA0OCwwLjA3ODIgLTAuMDQ4LDAuMTM3MzcgMCwwLjA1OTIgLTAuMDE4MSwwLjEyODEzIC0wLjA0MDIsMC4xNTMyMyAtMC4wMjIxLDAuMDI1MSAtMC4xMzMxOCwwLjMzNzQzIC0wLjI0NjY2LDAuNjk0MDggLTAuMTEzNiwwLjM1NjY1IC0wLjQ1MzMxLDEuMzgzMzcgLTAuNzU1MDgsMi4yODE2IC0xLjQ2OTk1LDQuMzc2MjIgLTEuMzg3MzMsNC4wODQ2NSAtMS4yNDA4Myw0LjM3ODE0IDAuMjA3NSwwLjQxNTk3IDAuNjkzODQsMC41OTkyMiAxLjA3NDc1LDAuNDA0OTIgMC4yMTE3MSwtMC4xMDgwNyAxMi40Mjk4OSwtMTIuMjk5OTUgMTIuNzM2MSwtMTIuNzA4NzIgMC4xMDE0NywtMC4xMzU0NSAwLjI3MjgzLC0wLjQ5NzM5IDAuMzI5NzUsLTAuNjk2MzcgMC4wMjY0LC0wLjA5MjUgMC4wNzI5LC0wLjI1NDkzIDAuMTAzMjcsLTAuMzYxMDkgMC4xMDAzOSwtMC4zNTAwNCAwLjAyMDUsLTEuMDgzNCAtMC4xNjk4LC0xLjU2MDI1IC0wLjE3OTI4LC0wLjQ0OTIzIC0wLjYwNDE0LC0wLjkwMDUxIC00LjcwMjYsLTQuOTk0MDUgLTIuMzEzNjYsLTIuMzEwNzcgLTQuMjA2NjYsLTQuMjM1NiAtNC4yMDY2NiwtNC4yNzcyNyAwLC0wLjA0MTggMC4wODQ1LC0wLjMyMzUgMC4xODc4MSwtMC42MjU5OSAwLjEwMzI3LC0wLjMwMjUgMC4yMzY0NCwtMC43MTIxIDAuMjk1NzcsLTAuOTEwMjQgMC4wNTk0LC0wLjE5ODE0IDAuMTI2MiwtMC4zODA3OSAwLjE0ODQyLC0wLjQwNTg4IDAuMDIyMywtMC4wMjUxIDAuMDQwNSwtMC4wOTQgMC4wNDA1LC0wLjE1MzIzIDAsLTAuMDU5MiAwLjAyMTYsLTAuMTIxMDUgMC4wNDgsLTAuMTM3MzggMC4wMjY0LC0wLjAxNjMgMC4wNDgsLTAuMDgxMiAwLjA0OCwtMC4xNDQxIDAsLTAuMDYyOSAwLjAyMTYsLTAuMTI3NzcgMC4wNDgsLTAuMTQ0MSAwLjAyNjQsLTAuMDE2MyAwLjA0OCwtMC4wODEyIDAuMDQ4LC0wLjE0NDEgMCwtMC4wNjI5IDAuMDIxNiwtMC4xMjc3NyAwLjA0OCwtMC4xNDQxIDAuMDI2NCwtMC4wMTYzIDAuMDQ4LC0wLjA4MTIgMC4wNDgsLTAuMTQ0MSAwLC0wLjA2MjkgMC4wMjE2LC0wLjEyNzc3IDAuMDQ4LC0wLjE0NDEgMC4wMjY0LC0wLjAxNjMgMC4wNDgsLTAuMDgxMiAwLjA0OCwtMC4xNDQxIDAsLTAuMDYyOSAwLjAyMTYsLTAuMTI3NzcgMC4wNDgsLTAuMTQ0MSAwLjAyNjQsLTAuMDE2MyAwLjA0OCwtMC4wODEyIDAuMDQ4LC0wLjE0NDEgMCwtMC4wNjI5IDAuMDIxNiwtMC4xMjc3NyAwLjA0OCwtMC4xNDQxIDAuMDI2NCwtMC4wMTYzIDAuMDQ4LC0wLjA4MTIgMC4wNDgsLTAuMTQ0MSAwLC0wLjA2MjkgMC4wMjE2LC0wLjEyNzc3IDAuMDQ4LC0wLjE0NDExIDAuMDI2NCwtMC4wMTYzIDAuMDQ4LC0wLjA4MTIgMC4wNDgsLTAuMTQ0MSAwLC0wLjA2MjkgMC4wMjE2LC0wLjEyNzc3IDAuMDQ4LC0wLjE0NDEgMC4wMjY0LC0wLjAxNjMgMC4wNDgsLTAuMDgxMiAwLjA0OCwtMC4xNDQxIDAsLTAuMDYyOSAwLjAyMTYsLTAuMTI3NzcgMC4wNDgsLTAuMTQ0MSAwLjAyNjQsLTAuMDE2MyAwLjA0OCwtMC4wODEyIDAuMDQ4LC0wLjE0NDEgMCwtMC4wNjI5IDAuMDIxNiwtMC4xMjc3NyAwLjA0OCwtMC4xNDQxIDAuMDI2NCwtMC4wMTYzIDAuMDQ4LC0wLjA4MTIgMC4wNDgsLTAuMTQ0MSAwLC0wLjA2MjkgMC4wMjE2LC0wLjEyNzc3IDAuMDQ4LC0wLjE0NDEgMC4wMjY0LC0wLjAxNjMgMC4wNDgsLTAuMDgxMiAwLjA0OCwtMC4xNDQxIDAsLTAuMDYyOSAwLjAyMTYsLTAuMTI3NzcgMC4wNDgsLTAuMTQ0MSAwLjAyNjQsLTAuMDE2MyAwLjA0OCwtMC4wODEyIDAuMDQ4LC0wLjE0NDEgMCwtMC4wNjI5IDAuMDIxNiwtMC4xMjc3NyAwLjA0OCwtMC4xNDQxIDAuMDI2NCwtMC4wMTYzIDAuMDQ4LC0wLjA4MTIgMC4wNDgsLTAuMTQ0MSAwLC0wLjA2MjkgMC4wMjE2LC0wLjEyNzc3IDAuMDQ4LC0wLjE0NDExIDAuMDI2NCwtMC4wMTYzIDAuMDQ4LC0wLjA4MTIgMC4wNDgsLTAuMTQ0MSAwLC0wLjA2MjkgMC4wMjE2LC0wLjEyNzc3IDAuMDQ4LC0wLjE0NDEgMC4wMjY0LC0wLjAxNjMgMC4wNDgsLTAuMDcxOSAwLjA0OCwtMC4xMjM1NiAwLC0wLjA1MTYgMC4wMTk1LC0wLjEzMTM3IDAuMDQzMiwtMC4xNzcyNSAwLjAyMzcsLTAuMDQ1OSAwLjM1NzYxLC0xLjAzNDUyIDAuNzQyLC0yLjE5NjkzIDAuMzg0MjcsLTEuMTYyNDEwMSAwLjg1MTI4LC0yLjU2MTc1MDEgMS4wMzc1MywtMy4xMDk4MTAxIDAuMzc4MDIsLTEuMTExNjIgMC4zOTQyMywtMS4yMjg0NiAwLjIxMDg3LC0xLjUxMTM4IC0wLjI0MDg5LC0wLjM3MTQyIC0wLjc1NzczLC0wLjUxNzMyIC0xLjA5Njk3LC0wLjMwOTgyIiBpZD0icGF0aDAiLz4KICA8cGF0aCBzdHlsZT0iZmlsbDojNTQwYzhjO2ZpbGwtcnVsZTpldmVub2RkO3N0cm9rZTpub25lO3N0cm9rZS13aWR0aDowLjEyMDA4MzkxIiBkPSJtIDAuMjMxNTczOSwwLjA1MjAwMTg2IGMgLTAuMDY1NiwwLjAzMzMgLTAuMTQ2MjYsMC4xMTM5NiAtMC4xNzk1MiwwLjE3OTUyIC0wLjA0ODMsMC4wOTUyIC0wLjA1ODEsNC44NzI2NTAwNCAtMC4wNDksMjMuODM0MzgwMTQgbCAwLjAxMTQsMjMuNzE1MjUgMC4xMDk0LDAuMTA5MzkgMC4xMDkzOSwwLjEwOTQgaCAyMy43NzM5NzAxIDIzLjc3Mzk4IGwgMC4xMDkzOSwtMC4xMDk0IDAuMTA5NCwtMC4xMDkzOSBWIDI0LjAwNzE3MiAwLjIzMzIwMTg2IGwgLTAuMTA5NCwtMC4xMDkzOSAtMC4xMDkzOSwtMC4xMDk0IC0yMy43MTUyNSwtMC4wMTE0IGMgLTE4Ljk2MTczMDEsLTAuMDA5IC0yMy43MzkxNTAxLDcuMmUtNCAtMjMuODM0MzgwMSwwLjA0OSBNIDI3LjU4MTI3NCw1LjAwNDYzMTkgYyAwLjc3OTcxLDAuMjcxMzkgMC44MjU1OCwwLjYzNzI4IDAuMjgwODgsMi4yMzg4NCAtMS4zMjQ2OCw0LjA4NzExMDEgLTIuNjk1NjUsOC4wNjUwNzAxIC0zLjg4ODgsMTEuNjY0MzUwMSBsIDQuMjczNTUsNC4yNjY5NCBjIDMuNzMyNDUsMy43MjYzMyA0LjU1MDQ2LDQuNTcwNjQgNC41NTA0Niw0LjY5NjI0IDAsMC4wMTU0IDAuMDM5OSwwLjExOTYxIDAuMDg4NiwwLjIzMTUzIDAuMzg3MzksMC44ODk4MiAwLjI2MzEsMS45OTQ5NSAtMC4zMDk5NCwyLjc1NjE2IC0wLjQyNDM3LDAuNTYzOCAtMTIuNjIyMDIsMTIuNjg1NDMgLTEyLjgyMDQsMTIuNzQwNTQgLTAuNDE4MjUsMC4xMTYxMyAtMC43NDU5NiwtMC4wMTggLTEuMDI5NzIsLTAuNDIxMjUgLTAuMjI5OTYsLTAuMzI2OTkgLTAuMjA5NTQsLTAuNDExNzcgMC44NjEwMSwtMy41ODM5IDEuMDM5ODksLTMuMzA1MjggMi4yNzcwMywtNi43Mjk2NSAzLjI2NzQ4LC05Ljg3MTk4IDAsLTAuMDIzNCAtMS45MDg0OSwtMS45NTExMyAtNC4yNDExMiwtNC4yODM3NiAtMy45ODAzMSwtMy45ODA0MiAtNC40Mjg4MiwtNC40NTA5MSAtNC41OTIxMywtNC44MTc1MiAtMC4wMzUzLC0wLjA3OTMgLTAuMDgzMiwtMC4xODE2OSAtMC4xMDY2NCwtMC4yMjc1NiAtMC4wMjMzLC0wLjA0NTkgLTAuMDQyNCwtMC4xMjI2MSAtMC4wNDI0LC0wLjE3MDUyIDAsLTAuMDQ3OSAtMC4wMjI3LC0wLjEwOTc2IC0wLjA1MDYsLTAuMTM3NjIgLTAuMDY3NiwtMC4wNjc2IC0wLjA2NzYsLTEuMzEwNzEgMCwtMS4zNzgzMiAwLjAyNzksLTAuMDI3OSAwLjA1MDYsLTAuMDk4MSAwLjA1MDYsLTAuMTU2MTEgMCwtMC4xMDg5MSAwLjIwNzUxLC0wLjUzNTA5IDAuMzkxMjQsLTAuODAzNzIgMC4yMzg5NiwtMC4zNDkyIDEyLjY2NTQ5LC0xMi43MTMwNDAxIDEyLjc3ODM3LC0xMi43MTM3NjAxIDAuMDYxNCwtMy42ZS00IDAuMTQ0MSwtMC4wMjE3IDAuMTgzNzIsLTAuMDQ3MyAwLjA5NDQsLTAuMDYxIDAuMTMyNDYsLTAuMDU5IDAuMzU1ODEsMC4wMTg3IiBpZD0icGF0aDEiLz4KPC9zdmc+Cg=="}}},{"cell_type":"code","source":"!pip install pytorch-lightning","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Importing libraries","metadata":{}},{"cell_type":"code","source":"import numpy as np \nimport pandas as pd \n\nimport os\nimport gc \nimport sys\n\nfrom IPython.core.display import display\nfrom ipywidgets import IntSlider, interact\nfrom matplotlib import pyplot as plt\nfrom mpl_toolkits.axes_grid1 import ImageGrid\n\nimport torch\nfrom torch.nn import functional as F\nfrom torch.utils.data import DataLoader\nfrom torch.utils.data import Dataset\nfrom torchvision import transforms, models\nfrom torch.optim.lr_scheduler import ReduceLROnPlateau\nfrom torch.optim import Adam\n\nfrom pytorch_lightning import Trainer\nfrom pytorch_lightning.logging import WandbLogger\nfrom pytorch_lightning.core import LightningModule\n\n\n# notebook params\n_ = plt.rcParams['figure.figsize'] = [15, 2]\nnp.random.seed(400)","metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"package_path = '../input/efficientnet/efficientnet-pytorch/EfficientNet-PyTorch/'\nsys.path.append(package_path)\nfrom efficientnet_pytorch import EfficientNet","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Data Overview","metadata":{}},{"cell_type":"code","source":"import os\nfor dirname, _, filenames in os.walk('/kaggle/input/bengaliai-cv19'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Reading The CSVs","metadata":{}},{"cell_type":"code","source":"class_map_df = pd.read_csv('/kaggle/input/bengaliai-cv19/class_map.csv')\nprint(f'class map shape: ', class_map_df.shape)\nclass_map_df.sample(50).drop_duplicates(['component_type'])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df = pd.read_csv('/kaggle/input/bengaliai-cv19/train.csv')\nprint(f'train data shape: ', train_df.shape)\nprint(f'unique graphemes: ', train_df['grapheme'].nunique())\nprint(f'unique grapheme_root: ', train_df['grapheme_root'].nunique())\nprint(f'unique vowel_diacritic: ', train_df['vowel_diacritic'].nunique())\nprint(f'unique consonant_diacritic: ', train_df['consonant_diacritic'].nunique())\n\ntrain_df.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df = pd.read_csv('/kaggle/input/bengaliai-cv19/test.csv')\nprint(f'test data shape: ', test_df.shape)\ntest_df.head(6)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_sub_df = pd.read_csv('/kaggle/input/bengaliai-cv19/sample_submission.csv')\nsample_sub_df.head(6)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# clean up\ndel train_df\ndel test_df\ndel class_map_df\n_ = gc.collect()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Reading The Image files\n\nSince loading the parquet files is a bit slow, we use another [public dataset](https://www.kaggle.com/corochann/bengaliaicv19feather) in [feather format](https://github.com/wesm/feather) which is around 30 times faster!","metadata":{"trusted":true}},{"cell_type":"markdown","source":"Some helper functions to handle and visualize the data","metadata":{}},{"cell_type":"code","source":"def get_image_data(mode='val', debug=False):\n    '''\n    helper function for PyTorch Dataset class\n    \n    Arguments:\n        mode (str) -- reads the feather files with train in the filename for train and val,\n                      and reads files with test in their names.\n                           \n    Returns:\n        img_df (dataframe) -- training images if train = true, else test images\n    '''\n    \n    img_list = []\n    file_type = mode # to fetch files\n    if mode == 'val':\n        file_type = 'train'\n    for dirname, _, filenames in os.walk('/kaggle/input/bengaliaicv19feather'):\n        for filename in filenames:\n            if file_type in filename:\n                img_list.append(pd.read_feather(os.path.join(dirname, filename)))\n                           \n    if mode == 'val':\n        img_df = pd.DataFrame(img_list[-1])\n    elif mode == 'train':\n        img_df = pd.concat(img_list[0:-1])\n    else:\n        img_df = pd.concat(img_list)\n        \n    print(f\"[Helper] {mode} image dataset: {img_df.shape}\")\n    \n    img_df = img_df[0:25]\n    \n    del img_list\n    _ = gc.collect()\n    \n    return img_df","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_images(rows, cols, img_df, train=True):\n    \"\"\"\n    Grid of images\n    \n    Arguments:\n        rows, cols (int, int) -- dimenstion of the image grid\n        img_df (dataframe) -- Dataframe of all the images\n        train (boolean) -- fetch meta data from the csv files accordingly\n    \"\"\"\n    \n    fig = plt.figure(figsize=(15., 12.))\n    grid = ImageGrid(fig, 111,  # similar to subplot(111)\n                 nrows_ncols=(rows, cols),  # creates 5x5 grid of axes\n                 axes_pad=0.3,  # pad between axes in inch.\n                 )\n    train_df = pd.read_csv('/kaggle/input/bengaliai-cv19/train.csv')\n    test_df = pd.read_csv('/kaggle/input/bengaliai-cv19/test.csv')\n\n    meta_df = train_df if train else test_df\n    \n    for ax, df_row in zip(grid, img_df.sample(rows*cols).values.tolist()):\n        # Iterating over the grid returns the Axes.\n        _ = ax.imshow(np.asarray(df_row[1:]).astype(int).reshape(137,236))\n        # fetch the sample's labels from the csv file\n        meta = meta_df[meta_df['image_id']==df_row[0]].values[0]\n        \n        if train:\n            title =  f'{df_row[0]}_{meta[1]}_{meta[2]}_{meta[3]}'\n        else: \n            title =  f'{df_row[0]}'\n            \n        _ = ax.set_title(title)\n        _ = ax.axis('off')\n   \n    _ = plt.show()\n    \n    del train_df\n    del test_df\n    del meta_df\n    _ = gc.collect()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n\n# get the data\nimg_df = get_image_data(mode='train', debug=True)\n# visualise few images \nplot_images(5, 5, img_df, train=True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# clear the data after visualization\n# Since we will be loading the data using PyTorch Dataset class again in the later cells\ndel img_df\n_ = gc.collect()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Preparing PyTorch Dataset","metadata":{}},{"cell_type":"code","source":"class BengaliAI(Dataset):\n    \"\"\"Bengali AI dataset for training PyTorch models\"\"\"\n\n    def __init__(self, mode='val', transform=None, debug=False):\n        \"\"\"\n        Arguments:\n            mode (str) -- to fetch appr. meta data. Default to val since smaller size\n            transform (callable) -- Transform to be applied on each sample \n        \"\"\"\n        self.mode = mode\n        if self.mode == 'train' or self.mode == 'val':         \n            self.metadata = pd.read_csv(f'/kaggle/input/bengaliai-cv19/train.csv')\n        else:\n            self.metadata = pd.read_csv(f'/kaggle/input/bengaliai-cv19/test.csv')\n\n        self.data = get_image_data(mode, debug)\n        self.transform = transform\n        \n        if self.mode != 'test':\n            _categorical_columns = ['grapheme_root', 'vowel_diacritic', 'consonant_diacritic']\n            self.grapheme_root = self.metadata[_categorical_columns[0]]\n            self.vowel_diacritic = self.metadata[_categorical_columns[1]]\n            self.constant_diacritic = self.metadata[_categorical_columns[2]]\n                                                \n    def __len__(self):\n        return len(self.data)\n    \n    def __getitem__(self, idx):\n        \"\"\"\n        Arguments:\n            idx (int) -- Dataset class is a map-style dataset (https://pytorch.org/docs/stable/data.html#map-style-datasets) \n                         where idx is the index of a specific sample in the map\n\n        Returns:\n            sample (dic) -- Each sample is a dic with keys as image_id, image, grapheme_root, vowel_diacritic, consonant_diacritic\n        \"\"\"\n        # we will disard the image id prefix 'Train_/Test_' so its easy to format it as tensor\n        # data = image data\n        _data_at_idx = self.data.iloc[idx]\n        _image_id = int(_data_at_idx[0].split('_')[1])\n        _image = _data_at_idx[1:]\n        \n        if self.mode == 'test':\n            sample = {'image_id': _image_id,\n              'image': _image\n             }\n        \n        else:\n            _grapheme_root = self.grapheme_root.iloc[idx]\n            _vowel_diacritic = self.vowel_diacritic.iloc[idx]\n            _constant_diacritic = self.constant_diacritic.iloc[idx]\n\n            sample = {'image_id': _image_id,\n                      'image': _image,\n                      'grapheme_root': _grapheme_root,\n                      'vowel_diacritic': _vowel_diacritic,\n                      'consonant_diacritic': _constant_diacritic\n                     }\n\n        if self.transform:\n            sample = self.transform(sample)\n            \n        return sample","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class ToTensor(object):\n    \"\"\"Convert ndarrays in sample to Tensors. And also makes it 3 channel\"\"\"\n\n    def __call__(self, sample):\n        for key in sample.keys():\n            sample[key] = torch.tensor(sample[key], dtype=torch.float32)\n            if key == 'image':\n                sample[key] = sample[key].reshape(137, 236).repeat(3, 1, 1)\n        \n        return sample","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Sanity Check","metadata":{}},{"cell_type":"code","source":"%%time \nbengali_dataset= BengaliAI(mode='test', transform=ToTensor(), debug=True)\nsample = bengali_dataset[0]\nsample","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del bengali_dataset\ngc.collect()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## The Lightning Module\nThis is \"**The Juice**\", where all your research and creativity goes!    \nPyTorch Lightning also provides [research seed](https://github.com/williamFalcon/pytorch-lightning-conference-seed), cookie-cutter-like template for your project repository.\n\n<div class=\"alert alert-block alert-info\">\n<b>💡</b>\nA [LightningModule](https://pytorch-lightning.readthedocs.io/en/latest/lightning-module.html) is a strict superclass of torch.nn.Module but provides an interface to standardize the “ingredients” for a research or production system. \n</div>","metadata":{}},{"cell_type":"code","source":"class ResNext3(LightningModule):\n\n    def __init__(self):\n        super(ResNext3, self).__init__()\n        # ResNext\n        # backbone_model = torch.hub.load('pytorch/vision:v0.5.0', 'resnext50_32x4d', pretrained=False)\n        self.toy_data = True\n        backbone_model = EfficientNet.from_name('efficientnet-b7') \n        backbone_model.load_state_dict(torch.load('../input/efficientnet-pytorch/efficientnet-b7-dcc49843.pth'))\n        # Take the whole resnext except for the last layer\n        backbone_layers = torch.nn.ModuleList(backbone_model.children())[:-2]\n        # Unpack all layers to Sequential as list is not a valid parameter \n        self.features = torch.nn.Sequential(*backbone_layers)\n        in_features = backbone_model._fc.in_features\n        self.fc_grapheme_root = torch.nn.Linear(in_features, 168)\n        self.fc_vowel_diacritic = torch.nn.Linear(in_features, 11)\n        self.fc_consonant_diacritic = torch.nn.Linear(in_features, 7)\n\n    def forward(self, x):\n        x = self.features(x)\n        x = torch.flatten(x,1)\n        grapheme = self.fc_grapheme_root(x)\n        vowel = self.fc_vowel_diacritic(x)\n        consonant = self.fc_consonant_diacritic(x)    \n        return grapheme, vowel, consonant\n        \n    def training_step(self, batch, batch_idx):\n        print(batch)\n        grapheme, vowel, consonant = self.forward(batch['image'])\n        loss_grapheme = F.cross_entropy(grapheme, batch['grapheme_root'].long())\n        loss_vowel = F.cross_entropy(vowel, batch['vowel_diacritic'].long())\n        loss_consonant = F.cross_entropy(consonant, batch['consonant_diacritic'].long())\n        logger_logs = {\"tl_grapheme\": loss_grapheme, \n                       \"tl_vowel\": loss_vowel, \n                       \"tl_consonant\": loss_consonant}\n        \n        return {'loss': loss_grapheme+loss_vowel+loss_consonant, 'log': logger_logs}\n\n    def validation_step(self, batch, batch_idx):\n        print(batch)\n        grapheme, vowel, consonant = self.forward(batch['image'])\n        loss_grapheme = F.cross_entropy(grapheme, batch['grapheme_root'].long())\n        loss_vowel = F.cross_entropy(vowel, batch['vowel_diacritic'].long())\n        loss_consonant = F.cross_entropy(consonant, batch['consonant_diacritic'].long())\n        logger_logs = {\"vl_grapheme\": loss_grapheme, \n                       \"vl_vowel\": loss_vowel, \n                       \"vl_consonant\": loss_consonant}\n        return {'val_loss': loss_grapheme+loss_vowel+loss_consonant, 'log': logger_logs}\n                                                                                            \n    def validation_end(self, outputs):\n        logger_logs = {'avg_val_loss': torch.stack([x['val_loss'] for x in outputs]).mean(),\n                       \"avl_grapheme\": torch.stack([x['log']['vl_grapheme'] for x in outputs]).mean(), \n                       \"avl_vowel\": torch.stack([x['log']['vl_vowel'] for x in outputs]).mean(), \n                       \"avl_consonant\": torch.stack([x['log']['vl_consonant'] for x in outputs]).mean()\n                       }\n        # must return 'val_loss' as key\n        return {'val_loss': logger_logs['avg_val_loss'], 'log': logger_logs} \n    \n    def configure_optimizers(self):\n        optimizer = Adam(self.parameters(), lr=0.001)\n        scheduler = ReduceLROnPlateau(optimizer)\n        return [optimizer],[scheduler]\n\n    def train_dataloader(self):\n        return DataLoader(BengaliAI(mode='train', transform=ToTensor()), batch_size=64, pin_memory=True)\n    \n    def val_dataloader(self):\n        return DataLoader(BengaliAI(mode='val', transform=ToTensor()), batch_size=64, pin_memory=True)        \n\n    def test_dataloader(self):\n        return DataLoader(BengaliAI(mode='test', transform=ToTensor()), batch_size=64, pin_memory=True)        ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = ResNext3()\nwandb_logger = WandbLogger(name='The-run', project='bengali-ai')\ntrainer = Trainer(gpus=0, fast_dev_run=True, checkpoint_callback=None)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- Pytorch Lightning supports many experiment tracking platforms like wandb, comet, test tube etc.   \n- The logger should be set to wandb_logger since it needs a key which can not be provided when commiting the kernel I set it to None.\n- Simply set gpus=1 or more to make it use gpus.  \n- Setting the fast_dev_run to True will run all steps of training once to make sure everything is in place.  \n\n  You have tons of these elegant abstractions, Isnt it cool?","metadata":{}},{"cell_type":"code","source":"trainer.fit(model)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"*This is a work in progress. Should I improve this, showing more features of PL or is it too late for example kernels for this competition?*","metadata":{}},{"cell_type":"code","source":"del trainer","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}