{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Bengali.AI SEResNeXt training with pytorch\n\nI will introduce following contents\n\n - **Fast data loading** with feather format\n - **Data augmentation** technic with affine transformation\n - **CNN SoTA models**: Use pytorch `pretrainedmodels` library, especially I use **`SEResNeXt`** in this notebook\n - **Training code abstraction**: Use `pytorch-ignite` module for the trainining abstraction\n \n### Update history\n\n - 2020/1/4 v2: Added albumentations augmentations introduced in [Bengali: albumentations data augmentation tutorial](https://www.kaggle.com/corochann/bengali-albumentations-data-augmentation-tutorial)"},{"metadata":{},"cell_type":"markdown","source":"# Table of Contents:\n**[Fast data loading with feather](#load)**<br>\n**[Dataset](#dataset)**<br>\n**[Data augmentation/processing](#processing)**<br>\n**[pytorch model & define classifier](#model)**<br>\n**[Training code](#train)**<br>\n**[Prediction](#pred)**<br>\n**[Reference and further reading](#ref)**<br>"},{"metadata":{},"cell_type":"markdown","source":"To install https://github.com/Cadene/pretrained-models.pytorch without internet connection, we can install library as \"dataset\".\n\nIt is uploaded by @rishabhiitbhu : https://www.kaggle.com/rishabhiitbhu/pretrainedmodels"},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install ../input/pretrainedmodels/pretrainedmodels-0.7.4/pretrainedmodels-0.7.4/ > /dev/null # no output","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"import gc\nimport os\nfrom pathlib import Path\nimport random\nimport sys\n\nfrom tqdm.notebook import tqdm\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nfrom IPython.core.display import display, HTML\n\n# --- plotly ---\nfrom plotly import tools, subplots\nimport plotly.offline as py\npy.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go\nimport plotly.express as px\nimport plotly.figure_factory as ff\n\n# --- models ---\nfrom sklearn import preprocessing\nfrom sklearn.model_selection import KFold\nimport lightgbm as lgb\nimport xgboost as xgb\nimport catboost as cb\n\n# --- setup ---\npd.set_option('max_columns', 50)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"debug=True\nsubmission=False\nbatch_size=32\ndevice='cuda:0'\nout='.'\nimage_size=64\narch='pretrained'\nmodel_name='se_resnext50_32x4d'","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":false,"_kg_hide-output":true},"cell_type":"code","source":"datadir = Path('/kaggle/input/bengaliai-cv19')\nfeatherdir = Path('/kaggle/input/bengaliaicv19feather')\noutdir = Path('.')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"# Read in the data CSV files\n# train = pd.read_csv(datadir/'train.csv')\n# test = pd.read_csv(datadir/'test.csv')\n# sample_submission = pd.read_csv(datadir/'sample_submission.csv')\n# class_map = pd.read_csv(datadir/'class_map.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Fast data loading with feather\n\nRefer [Bengali.AI super fast data loading with feather](https://www.kaggle.com/corochann/bengali-ai-super-fast-data-loading-with-feather) and [dataset](https://www.kaggle.com/corochann/bengaliaicv19feather) for detail.<br/>\nOriginal `parquet` format takes about 60 sec to load 1 data, while `feather` format takes about **2 sec to load 1 data!!!**\n\n### How to add dataset\n\nWhen you write kernel, click \"+ Add Data\" botton on right top.<br/>\nThen inside window pop-up, you can see \"Search Datasets\" text box on right top.<br/>\nYou can type \"bengaliai-cv19-feather\" to find this dataset and press \"Add\" botton to add the data."},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport gc\n\n\ndef prepare_image(datadir, featherdir, data_type='train',\n                  submission=False, indices=[0, 1, 2, 3]):\n    assert data_type in ['train', 'test']\n    if submission:\n        image_df_list = [pd.read_parquet(datadir / f'{data_type}_image_data_{i}.parquet')\n                         for i in indices]\n    else:\n        image_df_list = [pd.read_feather(featherdir / f'{data_type}_image_data_{i}.feather')\n                         for i in indices]\n\n    print('image_df_list', len(image_df_list))\n    HEIGHT = 137\n    WIDTH = 236\n    images = [df.iloc[:, 1:].values.reshape(-1, HEIGHT, WIDTH) for df in image_df_list]\n    del image_df_list\n    gc.collect()\n    images = np.concatenate(images, axis=0)\n    return images\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\ntrain = pd.read_csv(datadir/'train.csv')\ntrain_labels = train[['grapheme_root', 'vowel_diacritic', 'consonant_diacritic']].values\nindices = [0] if debug else [0, 1, 2, 3]\ntrain_images = prepare_image(\n    datadir, featherdir, data_type='train', submission=False, indices=indices)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"dataset\"></a>\n# Dataset"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"\"\"\"\nReferenced `chainer.dataset.DatasetMixin` to work with pytorch Dataset.\n\"\"\"\nimport numpy\nimport six\nimport torch\nfrom torch.utils.data.dataset import Dataset\n\n\nclass DatasetMixin(Dataset):\n\n    def __init__(self, transform=None):\n        self.transform = transform\n\n    def __getitem__(self, index):\n        \"\"\"Returns an example or a sequence of examples.\"\"\"\n        if torch.is_tensor(index):\n            index = index.tolist()\n        if isinstance(index, slice):\n            current, stop, step = index.indices(len(self))\n            return [self.get_example_wrapper(i) for i in\n                    six.moves.range(current, stop, step)]\n        elif isinstance(index, list) or isinstance(index, numpy.ndarray):\n            return [self.get_example_wrapper(i) for i in index]\n        else:\n            return self.get_example_wrapper(index)\n\n    def __len__(self):\n        \"\"\"Returns the number of data points.\"\"\"\n        raise NotImplementedError\n\n    def get_example_wrapper(self, i):\n        \"\"\"Wrapper of `get_example`, to apply `transform` if necessary\"\"\"\n        example = self.get_example(i)\n        if self.transform:\n            example = self.transform(example)\n        return example\n\n    def get_example(self, i):\n        \"\"\"Returns the i-th example.\n\n        Implementations should override it. It should raise :class:`IndexError`\n        if the index is invalid.\n\n        Args:\n            i (int): The index of the example.\n\n        Returns:\n            The i-th example.\n\n        \"\"\"\n        raise NotImplementedError\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This `DatasetMixin` class can be used to define any custom dataset class in pytorch. We can implement `get_example(self, i)` method to return `i`-th data.\n\nHere I return i-th image `x` and `label`, with scaling image to be value ranges between 0~1."},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\n\n\nclass BengaliAIDataset(DatasetMixin):\n    def __init__(self, images, labels=None, transform=None, indices=None):\n        super(BengaliAIDataset, self).__init__(transform=transform)\n        self.images = images\n        self.labels = labels\n        if indices is None:\n            indices = np.arange(len(images))\n        self.indices = indices\n        self.train = labels is not None\n\n    def __len__(self):\n        \"\"\"return length of this dataset\"\"\"\n        return len(self.indices)\n\n    def get_example(self, i):\n        \"\"\"Return i-th data\"\"\"\n        i = self.indices[i]\n        x = self.images[i]\n        # Opposite white and black: background will be white and\n        # for future Affine transformation\n        x = (255 - x).astype(np.float32) / 255.\n        if self.train:\n            y = self.labels[i]\n            return x, y\n        else:\n            return x\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's see how this `BengaliAIDataset` work"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_dataset = BengaliAIDataset(train_images, train_labels)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`train_dataset[i]` returns i-th image array and 3 target labels (graphme_root, vowel_diacritic and consonant_diacritic)."},{"metadata":{"trusted":true},"cell_type":"code","source":"image, label = train_dataset[0]\nprint('image', image.shape, 'label', label)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"nrow, ncol = 5, 6\n\nfig, axes = plt.subplots(nrow, ncol, figsize=(20, 8))\naxes = axes.flatten()\nfor i, ax in tqdm(enumerate(axes)):\n    image, label = train_dataset[i]\n    ax.imshow(image, cmap='Greys')\n    ax.set_title(f'label: {label}')\nplt.tight_layout()\nplt.show()\nplt.savefig('bengaliai.png')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"processing\"></a>\n# Data augmentation/processing"},{"metadata":{},"cell_type":"markdown","source":"For CNN training, data augmentation is important to improve test accuracy (generalization performance). I will show some image preprocessing to increase the data variety."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"\"\"\"\nFrom https://www.kaggle.com/corochann/deep-learning-cnn-with-chainer-lb-0-99700\n\"\"\"\nimport cv2\nfrom skimage.transform import AffineTransform, warp\nimport numpy as np\n\n\ndef affine_image(img):\n    \"\"\"\n\n    Args:\n        img: (h, w) or (1, h, w)\n\n    Returns:\n        img: (h, w)\n    \"\"\"\n    # ch, h, w = img.shape\n    # img = img / 255.\n    if img.ndim == 3:\n        img = img[0]\n\n    # --- scale ---\n    min_scale = 0.8\n    max_scale = 1.2\n    sx = np.random.uniform(min_scale, max_scale)\n    sy = np.random.uniform(min_scale, max_scale)\n\n    # --- rotation ---\n    max_rot_angle = 7\n    rot_angle = np.random.uniform(-max_rot_angle, max_rot_angle) * np.pi / 180.\n\n    # --- shear ---\n    max_shear_angle = 10\n    shear_angle = np.random.uniform(-max_shear_angle, max_shear_angle) * np.pi / 180.\n\n    # --- translation ---\n    max_translation = 4\n    tx = np.random.randint(-max_translation, max_translation)\n    ty = np.random.randint(-max_translation, max_translation)\n\n    tform = AffineTransform(scale=(sx, sy), rotation=rot_angle, shear=shear_angle,\n                            translation=(tx, ty))\n    transformed_image = warp(img, tform)\n    assert transformed_image.ndim == 2\n    return transformed_image\n\n\ndef crop_char_image(image, threshold=5./255.):\n    assert image.ndim == 2\n    is_black = image > threshold\n\n    is_black_vertical = np.sum(is_black, axis=0) > 0\n    is_black_horizontal = np.sum(is_black, axis=1) > 0\n    left = np.argmax(is_black_horizontal)\n    right = np.argmax(is_black_horizontal[::-1])\n    top = np.argmax(is_black_vertical)\n    bottom = np.argmax(is_black_vertical[::-1])\n    height, width = image.shape\n    cropped_image = image[left:height - right, top:width - bottom]\n    return cropped_image\n\n\ndef resize(image, size=(128, 128)):\n    return cv2.resize(image, size)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Affine transformation for data augmentation\n\nTo increase validation score, the number of training data is important. When we can use more number of training data, we can reduce overfitting and validation score becomes high.\n\n\"Data augmentation\" is a technic to virtually create extra training data, based on the given training data. For this MNIST task, data augmentation can be achieved by utilizing affine transformation.\n\n1. Rotation AffineTransformation\n2. Translation\n3. Scale\n4. Shear"},{"metadata":{"trusted":true},"cell_type":"code","source":"nrow, ncol = 1, 6\n\nfig, axes = plt.subplots(nrow, ncol, figsize=(20, 8))\naxes = axes.flatten()\nfor i, ax in tqdm(enumerate(axes)):\n    image, label = train_dataset[0]\n    ax.imshow(affine_image(image), cmap='Greys')\n    ax.set_title(f'label: {label}')\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"When the image is slightly rotated, shifted (transformed) or scaled, the image looks like the same label. We can virtually create another image data from one image in such a way."},{"metadata":{},"cell_type":"markdown","source":"## crop image\n\nHere I crop image"},{"metadata":{"trusted":true},"cell_type":"code","source":"nrow, ncol = 5, 6\n\nfig, axes = plt.subplots(nrow, ncol, figsize=(20, 8))\naxes = axes.flatten()\nfor i, ax in tqdm(enumerate(axes)):\n    image, label = train_dataset[i]\n    ax.imshow(crop_char_image(image, threshold=20./255.), cmap='Greys')\n    ax.set_title(f'label: {label}')\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## resize image\n\nWe need to resize image after crop, to align image size for CNN batch training."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"nrow, ncol = 5, 6\n\nfig, axes = plt.subplots(nrow, ncol, figsize=(20, 8))\naxes = axes.flatten()\nfor i, ax in tqdm(enumerate(axes)):\n    image, label = train_dataset[i]\n    ax.imshow(resize(crop_char_image(image, threshold=20./255.)), cmap='Greys')\n    ax.set_title(f'label: {label}')\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Put everything together with `Transform` class. <br>\n[Update] I added **albumentations augmentations** introduced in [Bengali: albumentations data augmentation tutorial](https://www.kaggle.com/corochann/bengali-albumentations-data-augmentation-tutorial)."},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"import albumentations as A\nimport numpy as np\n\n\ndef add_gaussian_noise(x, sigma):\n    x += np.random.randn(*x.shape) * sigma\n    x = np.clip(x, 0., 1.)\n    return x\n\n\ndef _evaluate_ratio(ratio):\n    if ratio <= 0.:\n        return False\n    return np.random.uniform() < ratio\n\n\ndef apply_aug(aug, image):\n    return aug(image=image)['image']\n\n\nclass Transform:\n    def __init__(self, affine=True, crop=True, size=(64, 64),\n                 normalize=True, train=True, threshold=40.,\n                 sigma=-1., blur_ratio=0., noise_ratio=0., cutout_ratio=0.,\n                 grid_distortion_ratio=0., elastic_distortion_ratio=0., random_brightness_ratio=0.,\n                 piece_affine_ratio=0., ssr_ratio=0.):\n        self.affine = affine\n        self.crop = crop\n        self.size = size\n        self.normalize = normalize\n        self.train = train\n        self.threshold = threshold / 255.\n        self.sigma = sigma / 255.\n\n        self.blur_ratio = blur_ratio\n        self.noise_ratio = noise_ratio\n        self.cutout_ratio = cutout_ratio\n        self.grid_distortion_ratio = grid_distortion_ratio\n        self.elastic_distortion_ratio = elastic_distortion_ratio\n        self.random_brightness_ratio = random_brightness_ratio\n        self.piece_affine_ratio = piece_affine_ratio\n        self.ssr_ratio = ssr_ratio\n\n    def __call__(self, example):\n        if self.train:\n            x, y = example\n        else:\n            x = example\n        # --- Augmentation ---\n        if self.affine:\n            x = affine_image(x)\n\n        # --- Train/Test common preprocessing ---\n        if self.crop:\n            x = crop_char_image(x, threshold=self.threshold)\n        if self.size is not None:\n            x = resize(x, size=self.size)\n        if self.sigma > 0.:\n            x = add_gaussian_noise(x, sigma=self.sigma)\n\n        # albumentations...\n        x = x.astype(np.float32)\n        assert x.ndim == 2\n        # 1. blur\n        if _evaluate_ratio(self.blur_ratio):\n            r = np.random.uniform()\n            if r < 0.25:\n                x = apply_aug(A.Blur(p=1.0), x)\n            elif r < 0.5:\n                x = apply_aug(A.MedianBlur(blur_limit=5, p=1.0), x)\n            elif r < 0.75:\n                x = apply_aug(A.GaussianBlur(p=1.0), x)\n            else:\n                x = apply_aug(A.MotionBlur(p=1.0), x)\n\n        if _evaluate_ratio(self.noise_ratio):\n            r = np.random.uniform()\n            if r < 0.50:\n                x = apply_aug(A.GaussNoise(var_limit=5. / 255., p=1.0), x)\n            else:\n                x = apply_aug(A.MultiplicativeNoise(p=1.0), x)\n\n        if _evaluate_ratio(self.cutout_ratio):\n            # A.Cutout(num_holes=2,  max_h_size=2, max_w_size=2, p=1.0)  # Deprecated...\n            x = apply_aug(A.CoarseDropout(max_holes=8, max_height=8, max_width=8, p=1.0), x)\n\n        if _evaluate_ratio(self.grid_distortion_ratio):\n            x = apply_aug(A.GridDistortion(p=1.0), x)\n\n        if _evaluate_ratio(self.elastic_distortion_ratio):\n            x = apply_aug(A.ElasticTransform(\n                sigma=50, alpha=1, alpha_affine=10, p=1.0), x)\n\n        if _evaluate_ratio(self.random_brightness_ratio):\n            # A.RandomBrightness(p=1.0)  # Deprecated...\n            # A.RandomContrast(p=1.0)    # Deprecated...\n            x = apply_aug(A.RandomBrightnessContrast(p=1.0), x)\n\n        if _evaluate_ratio(self.piece_affine_ratio):\n            x = apply_aug(A.IAAPiecewiseAffine(p=1.0), x)\n\n        if _evaluate_ratio(self.ssr_ratio):\n            x = apply_aug(A.ShiftScaleRotate(\n                shift_limit=0.0625,\n                scale_limit=0.1,\n                rotate_limit=30,\n                p=1.0), x)\n\n        if self.normalize:\n            x = (x.astype(np.float32) - 0.0692) / 0.2051\n        if x.ndim == 2:\n            x = x[None, :, :]\n        x = x.astype(np.float32)\n        if self.train:\n            y = y.astype(np.int64)\n            return x, y\n        else:\n            return x","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_transform = Transform(\n    size=(image_size, image_size), threshold=20.,\n    sigma=-1., blur_ratio=0.2, noise_ratio=0.2, cutout_ratio=0.2,\n    grid_distortion_ratio=0.2, random_brightness_ratio=0.2,\n    piece_affine_ratio=0.2, ssr_ratio=0.2)\n# transform = Transform(size=(image_size, image_size)\ntrain_dataset = BengaliAIDataset(train_images, train_labels,\n                                 transform=train_transform)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"By setting `transform`, its function is called **every time** when we access to the index. Dataset returns different `image` every time! which is useful for training with data augmentation."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"nrow, ncol = 1, 6\n\nfig, axes = plt.subplots(nrow, ncol, figsize=(20, 2))\naxes = axes.flatten()\nfor i, ax in tqdm(enumerate(axes)):\n    image, label = train_dataset[0]\n    ax.imshow(image[0], cmap='Greys')\n    ax.set_title(f'label: {label}')\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's final check the processed images, which will be trained by the model."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"nrow, ncol = 5, 6\n\nfig, axes = plt.subplots(nrow, ncol, figsize=(20, 8))\naxes = axes.flatten()\nfor i, ax in tqdm(enumerate(axes)):\n    image, label = train_dataset[i]\n    ax.imshow(image[0], cmap='Greys')\n    ax.set_title(f'label: {label}')\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"model\"></a> \n# pytorch model & define classifier"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import torch\n\n\ndef residual_add(lhs, rhs):\n    lhs_ch, rhs_ch = lhs.shape[1], rhs.shape[1]\n    if lhs_ch < rhs_ch:\n        out = lhs + rhs[:, :lhs_ch]\n    elif lhs_ch > rhs_ch:\n        out = torch.cat([lhs[:, :rhs_ch] + rhs, lhs[:, rhs_ch:]], dim=1)\n    else:\n        out = lhs + rhs\n    return out\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"from typing import List\n\nimport torch\nfrom torch import nn\nfrom torch.nn.parameter import Parameter\n\n\nclass LazyLoadModule(nn.Module):\n    \"\"\"Lazy buffer/parameter loading using load_state_dict_pre_hook\n\n    Define all buffer/parameter in `_lazy_buffer_keys`/`_lazy_parameter_keys` and\n    save buffer with `register_buffer`/`register_parameter`\n    method, which can be outside of __init__ method.\n    Then this module can load any shape of Tensor during de-serializing.\n\n    Note that default value of lazy buffer is torch.Tensor([]), while lazy parameter is None.\n    \"\"\"\n    _lazy_buffer_keys: List[str] = []     # It needs to be override to register lazy buffer\n    _lazy_parameter_keys: List[str] = []  # It needs to be override to register lazy parameter\n\n    def __init__(self):\n        super(LazyLoadModule, self).__init__()\n        for k in self._lazy_buffer_keys:\n            self.register_buffer(k, torch.tensor([]))\n        for k in self._lazy_parameter_keys:\n            self.register_parameter(k, None)\n        self._register_load_state_dict_pre_hook(self._hook)\n\n    def _hook(self, state_dict, prefix, local_metadata, strict, missing_keys,\n             unexpected_keys, error_msgs):\n        for key in self._lazy_buffer_keys:\n            self.register_buffer(key, state_dict[prefix + key])\n\n        for key in self._lazy_parameter_keys:\n            self.register_parameter(key, Parameter(state_dict[prefix + key]))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import math\nimport torch\nfrom torch.nn import init\nfrom torch.nn.parameter import Parameter\nimport torch.nn.functional as F\n\n\nclass LazyLinear(LazyLoadModule):\n    \"\"\"Linear module with lazy input inference\n\n    `in_features` can be `None`, and it is determined at the first time of forward step dynamically.\n    \"\"\"\n\n    __constants__ = ['bias', 'in_features', 'out_features']\n    _lazy_parameter_keys = ['weight']\n\n    def __init__(self, in_features, out_features, bias=True):\n        super(LazyLinear, self).__init__()\n        self.in_features = in_features\n        self.out_features = out_features\n        if bias:\n            self.bias = Parameter(torch.Tensor(out_features))\n        else:\n            self.register_parameter('bias', None)\n\n        if in_features is not None:\n            self.weight = Parameter(torch.Tensor(out_features, in_features))\n            self.reset_parameters()\n\n    def reset_parameters(self):\n        init.kaiming_uniform_(self.weight, a=math.sqrt(5))\n        if self.bias is not None:\n            fan_in, _ = init._calculate_fan_in_and_fan_out(self.weight)\n            bound = 1 / math.sqrt(fan_in)\n            init.uniform_(self.bias, -bound, bound)\n\n    def forward(self, input):\n        if self.weight is None:\n            self.in_features = input.shape[-1]\n            self.weight = Parameter(torch.Tensor(self.out_features, self.in_features))\n            self.reset_parameters()\n\n            # Need to send lazy defined parameter to device...\n            self.to(input.device)\n        return F.linear(input, self.weight, self.bias)\n\n    def extra_repr(self):\n        return 'in_features={}, out_features={}, bias={}'.format(\n            self.in_features, self.out_features, self.bias is not None\n        )\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"from torch import nn\nimport torch.nn.functional as F\n\n\nclass LinearBlock(nn.Module):\n\n    def __init__(self, in_features, out_features, bias=True,\n                 use_bn=True, activation=F.relu, dropout_ratio=-1, residual=False,):\n        super(LinearBlock, self).__init__()\n        if in_features is None:\n            self.linear = LazyLinear(in_features, out_features, bias=bias)\n        else:\n            self.linear = nn.Linear(in_features, out_features, bias=bias)\n        if use_bn:\n            self.bn = nn.BatchNorm1d(out_features)\n        if dropout_ratio > 0.:\n            self.dropout = nn.Dropout(p=dropout_ratio)\n        else:\n            self.dropout = None\n        self.activation = activation\n        self.use_bn = use_bn\n        self.dropout_ratio = dropout_ratio\n        self.residual = residual\n\n    def __call__(self, x):\n        h = self.linear(x)\n        if self.use_bn:\n            h = self.bn(h)\n        if self.activation is not None:\n            h = self.activation(h)\n        if self.residual:\n            h = residual_add(h, x)\n        if self.dropout_ratio > 0:\n            h = self.dropout(h)\n        return h","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import pretrainedmodels\nimport torch\nfrom torch import nn\nimport torch.nn.functional as F\nfrom torch.nn import Sequential\n\n\nclass PretrainedCNN(nn.Module):\n    def __init__(self, model_name='se_resnext101_32x4d',\n                 in_channels=1, out_dim=10, use_bn=True,\n                 pretrained='imagenet'):\n        super(PretrainedCNN, self).__init__()\n        self.conv0 = nn.Conv2d(\n            in_channels, 3, kernel_size=3, stride=1, padding=1, bias=True)\n        self.base_model = pretrainedmodels.__dict__[model_name](pretrained=pretrained)\n        activation = F.leaky_relu\n        self.do_pooling = True\n        if self.do_pooling:\n            inch = self.base_model.last_linear.in_features\n        else:\n            inch = None\n        hdim = 512\n        lin1 = LinearBlock(inch, hdim, use_bn=use_bn, activation=activation, residual=False)\n        lin2 = LinearBlock(hdim, out_dim, use_bn=use_bn, activation=None, residual=False)\n        self.lin_layers = Sequential(lin1, lin2)\n\n    def forward(self, x):\n        h = self.conv0(x)\n        h = self.base_model.features(h)\n\n        if self.do_pooling:\n            h = torch.sum(h, dim=(-1, -2))\n        else:\n            # [128, 2048, 4, 4] when input is (128, 128)\n            bs, ch, height, width = h.shape\n            h = h.view(bs, ch*height*width)\n        for layer in self.lin_layers:\n            h = layer(h)\n        return h","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Classifier"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import torch\nfrom torch import nn\nimport torch.nn.functional as F\nfrom tqdm import tqdm\n\n\ndef accuracy(y, t):\n    pred_label = torch.argmax(y, dim=1)\n    count = pred_label.shape[0]\n    correct = (pred_label == t).sum().type(torch.float32)\n    acc = correct / count\n    return acc\n\n\nclass BengaliClassifier(nn.Module):\n    def __init__(self, predictor, n_grapheme=168, n_vowel=11, n_consonant=7):\n        super(BengaliClassifier, self).__init__()\n        self.n_grapheme = n_grapheme\n        self.n_vowel = n_vowel\n        self.n_consonant = n_consonant\n        self.n_total_class = self.n_grapheme + self.n_vowel + self.n_consonant\n        self.predictor = predictor\n\n        self.metrics_keys = [\n            'loss', 'loss_grapheme', 'loss_vowel', 'loss_consonant',\n            'acc_grapheme', 'acc_vowel', 'acc_consonant']\n\n    def forward(self, x, y=None):\n        pred = self.predictor(x)\n        if isinstance(pred, tuple):\n            assert len(pred) == 3\n            preds = pred\n        else:\n            assert pred.shape[1] == self.n_total_class\n            preds = torch.split(pred, [self.n_grapheme, self.n_vowel, self.n_consonant], dim=1)\n        loss_grapheme = F.cross_entropy(preds[0], y[:, 0])\n        loss_vowel = F.cross_entropy(preds[1], y[:, 1])\n        loss_consonant = F.cross_entropy(preds[2], y[:, 2])\n        loss = loss_grapheme + loss_vowel + loss_consonant\n        metrics = {\n            'loss': loss.item(),\n            'loss_grapheme': loss_grapheme.item(),\n            'loss_vowel': loss_vowel.item(),\n            'loss_consonant': loss_consonant.item(),\n            'acc_grapheme': accuracy(preds[0], y[:, 0]),\n            'acc_vowel': accuracy(preds[1], y[:, 1]),\n            'acc_consonant': accuracy(preds[2], y[:, 2]),\n        }\n        return loss, metrics, pred\n\n    def calc(self, data_loader):\n        device: torch.device = next(self.parameters()).device\n        self.eval()\n        output_list = []\n        with torch.no_grad():\n            for batch in tqdm(data_loader):\n                # TODO: support general preprocessing.\n                # If `data` is not `Data` instance, `to` method is not supported!\n                batch = batch.to(device)\n                pred = self.predictor(batch)\n                output_list.append(pred)\n        output = torch.cat(output_list, dim=0)\n        preds = torch.split(output, [self.n_grapheme, self.n_vowel, self.n_consonant], dim=1)\n        return preds\n\n    def predict_proba(self, data_loader):\n        preds = self.calc(data_loader)\n        return [F.softmax(p, dim=1) for p in preds]\n\n    def predict(self, data_loader):\n        preds = self.calc(data_loader)\n        pred_labels = [torch.argmax(p, dim=1) for p in preds]\n        return pred_labels\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"train\"></a>\n# Training code"},{"metadata":{},"cell_type":"markdown","source":"## prepare data"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"n_dataset = len(train_images)\ntrain_data_size = 200 if debug else int(n_dataset * 0.9)\nvalid_data_size = 100 if debug else int(n_dataset - train_data_size)\n\nperm = np.random.RandomState(777).permutation(n_dataset)\nprint('perm', perm)\ntrain_dataset = BengaliAIDataset(\n    train_images, train_labels, transform=Transform(size=(image_size, image_size)),\n    indices=perm[:train_data_size])\nvalid_dataset = BengaliAIDataset(\n    train_images, train_labels, transform=Transform(affine=False, crop=True, size=(image_size, image_size)),\n    indices=perm[train_data_size:train_data_size+valid_data_size])\nprint('train_dataset', len(train_dataset), 'valid_dataset', len(valid_dataset))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# --- Model ---\ndevice = torch.device(device)\nn_grapheme = 168\nn_vowel = 11\nn_consonant = 7\nn_total = n_grapheme + n_vowel + n_consonant\nprint('n_total', n_total)\n# Set pretrained='imagenet' to download imagenet pretrained model...\npredictor = PretrainedCNN(in_channels=1, out_dim=n_total, model_name=model_name, pretrained=None)\nprint('predictor', type(predictor))\n\nclassifier = BengaliClassifier(predictor).to(device)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Ignite utility\n\npytorch-ignite utility class for training"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import json\nfrom logging import getLogger\nimport numpy\n\ndef save_json(filepath, params):\n    with open(filepath, 'w') as f:\n        json.dump(params, f, indent=4)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import os\nfrom logging import getLogger\nfrom time import perf_counter\n\nimport pandas as pd\nimport torch\n# from chainer_chemistry.utils import save_json\n\nfrom ignite.engine.engine import Engine, Events\nfrom ignite.metrics import Average\n\n\nclass DictOutputTransform:\n    def __init__(self, key, index=0):\n        self.key = key\n        self.index = index\n\n    def __call__(self, x):\n        if self.index >= 0:\n            x = x[self.index]\n        return x[self.key]\n\n\ndef create_trainer(classifier, optimizer, device):\n    classifier.to(device)\n\n    def update_fn(engine, batch):\n        classifier.train()\n        optimizer.zero_grad()\n        # batch = [elem.to(device) for elem in batch]\n        x, y = [elem.to(device) for elem in batch]\n        loss, metrics, pred_y = classifier(x, y)\n        loss.backward()\n        optimizer.step()\n        return metrics, pred_y, y\n    trainer = Engine(update_fn)\n\n    for key in classifier.metrics_keys:\n        Average(output_transform=DictOutputTransform(key)).attach(trainer, key)\n    return trainer\n\n\ndef create_evaluator(classifier, device):\n    classifier.to(device)\n\n    def update_fn(engine, batch):\n        classifier.eval()\n        with torch.no_grad():\n            # batch = [elem.to(device) for elem in batch]\n            x, y = [elem.to(device) for elem in batch]\n            _, metrics, pred_y = classifier(x, y)\n            return metrics, pred_y, y\n    evaluator = Engine(update_fn)\n\n    for key in classifier.metrics_keys:\n        Average(output_transform=DictOutputTransform(key)).attach(evaluator, key)\n    return evaluator\n\n\nclass LogReport:\n    def __init__(self, evaluator=None, dirpath=None, logger=None):\n        self.evaluator = evaluator\n        self.dirpath = str(dirpath) if dirpath is not None else None\n        self.logger = logger or getLogger(__name__)\n\n        self.reported_dict = {}  # To handle additional parameter to monitor\n        self.history = []\n        self.start_time = perf_counter()\n\n    def report(self, key, value):\n        self.reported_dict[key] = value\n\n    def __call__(self, engine):\n        elapsed_time = perf_counter() - self.start_time\n        elem = {'epoch': engine.state.epoch,\n                'iteration': engine.state.iteration}\n        elem.update({f'train/{key}': value\n                     for key, value in engine.state.metrics.items()})\n        if self.evaluator is not None:\n            elem.update({f'valid/{key}': value\n                         for key, value in self.evaluator.state.metrics.items()})\n        elem.update(self.reported_dict)\n        elem['elapsed_time'] = elapsed_time\n        self.history.append(elem)\n        if self.dirpath:\n            save_json(os.path.join(self.dirpath, 'log.json'), self.history)\n            self.get_dataframe().to_csv(os.path.join(self.dirpath, 'log.csv'), index=False)\n\n        # --- print ---\n        msg = ''\n        for key, value in elem.items():\n            if key in ['iteration']:\n                # skip printing some parameters...\n                continue\n            elif isinstance(value, int):\n                msg += f'{key} {value: >6d} '\n            else:\n                msg += f'{key} {value: 8f} '\n#         self.logger.warning(msg)\n        print(msg)\n\n        # --- Reset ---\n        self.reported_dict = {}\n\n    def get_dataframe(self):\n        df = pd.DataFrame(self.history)\n        return df\n\n\nclass SpeedCheckHandler:\n    def __init__(self, iteration_interval=10, logger=None):\n        self.iteration_interval = iteration_interval\n        self.logger = logger or getLogger(__name__)\n        self.prev_time = perf_counter()\n\n    def __call__(self, engine: Engine):\n        if engine.state.iteration % self.iteration_interval == 0:\n            cur_time = perf_counter()\n            spd = self.iteration_interval / (cur_time - self.prev_time)\n            self.logger.warning(f'{spd} iter/sec')\n            # reset\n            self.prev_time = cur_time\n\n    def attach(self, engine: Engine):\n        engine.add_event_handler(Events.ITERATION_COMPLETED, self)\n\n\nclass ModelSnapshotHandler:\n    def __init__(self, model, filepath='model_{count:06}.pt',\n                 interval=1, logger=None):\n        self.model = model\n        self.filepath: str = str(filepath)\n        self.interval = interval\n        self.logger = logger or getLogger(__name__)\n        self.count = 0\n\n    def __call__(self, engine: Engine):\n        self.count += 1\n        if self.count % self.interval == 0:\n            filepath = self.filepath.format(count=self.count)\n            torch.save(self.model.state_dict(), filepath)\n            # self.logger.warning(f'save model to {filepath}...')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import warnings\n\nimport torch\n\nfrom ignite.metrics.metric import Metric\n\n\nclass EpochMetric(Metric):\n    \"\"\"Class for metrics that should be computed on the entire output history of a model.\n    Model's output and targets are restricted to be of shape `(batch_size, n_classes)`. Output\n    datatype should be `float32`. Target datatype should be `long`.\n\n    .. warning::\n\n        Current implementation stores all input data (output and target) in as tensors before computing a metric.\n        This can potentially lead to a memory error if the input data is larger than available RAM.\n\n\n    - `update` must receive output of the form `(y_pred, y)`.\n\n    If target shape is `(batch_size, n_classes)` and `n_classes > 1` than it should be binary: e.g. `[[0, 1, 0, 1], ]`.\n\n    Args:\n        compute_fn (callable): a callable with the signature (`torch.tensor`, `torch.tensor`) takes as the input\n            `predictions` and `targets` and returns a scalar.\n        output_transform (callable, optional): a callable that is used to transform the\n            :class:`~ignite.engine.Engine`'s `process_function`'s output into the\n            form expected by the metric. This can be useful if, for example, you have a multi-output model and\n            you want to compute the metric with respect to one of the outputs.\n\n    \"\"\"\n\n    def __init__(self, compute_fn, output_transform=lambda x: x):\n\n        if not callable(compute_fn):\n            raise TypeError(\"Argument compute_fn should be callable.\")\n\n        super(EpochMetric, self).__init__(output_transform=output_transform)\n        self.compute_fn = compute_fn\n\n    def reset(self):\n        self._predictions = torch.tensor([], dtype=torch.float32)\n        self._targets = torch.tensor([], dtype=torch.long)\n\n    def update(self, output):\n        y_pred, y = output\n        self._predictions = torch.cat([self._predictions, y_pred], dim=0)\n        self._targets = torch.cat([self._targets, y], dim=0)\n\n        # Check once the signature and execution of compute_fn\n        if self._predictions.shape == y_pred.shape:\n            try:\n                self.compute_fn(self._predictions, self._targets)\n            except Exception as e:\n                warnings.warn(\"Probably, there can be a problem with `compute_fn`:\\n {}.\".format(e),\n                              RuntimeWarning)\n\n    def compute(self):\n        return self.compute_fn(self._predictions, self._targets)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import numpy as np\nimport sklearn.metrics\nimport torch\n\n\ndef macro_recall(pred_y, y, n_grapheme=168, n_vowel=11, n_consonant=7):\n    pred_y = torch.split(pred_y, [n_grapheme, n_vowel, n_consonant], dim=1)\n    pred_labels = [torch.argmax(py, dim=1).cpu().numpy() for py in pred_y]\n\n    y = y.cpu().numpy()\n    # pred_y = [p.cpu().numpy() for p in pred_y]\n\n    recall_grapheme = sklearn.metrics.recall_score(pred_labels[0], y[:, 0], average='macro')\n    recall_vowel = sklearn.metrics.recall_score(pred_labels[1], y[:, 1], average='macro')\n    recall_consonant = sklearn.metrics.recall_score(pred_labels[2], y[:, 2], average='macro')\n    scores = [recall_grapheme, recall_vowel, recall_consonant]\n    final_score = np.average(scores, weights=[2, 1, 1])\n    # print(f'recall: grapheme {recall_grapheme}, vowel {recall_vowel}, consonant {recall_consonant}, '\n    #       f'total {final_score}, y {y.shape}')\n    return final_score\n\n\ndef calc_macro_recall(solution, submission):\n    # solution df, submission df\n    scores = []\n    for component in ['grapheme_root', 'consonant_diacritic', 'vowel_diacritic']:\n        y_true_subset = solution[solution[component] == component]['target'].values\n        y_pred_subset = submission[submission[component] == component]['target'].values\n        scores.append(sklearn.metrics.recall_score(\n            y_true_subset, y_pred_subset, average='macro'))\n    final_score = np.average(scores, weights=[2, 1, 1])\n    return final_score\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"import argparse\nfrom distutils.util import strtobool\nimport os\n\nimport torch\nfrom ignite.contrib.handlers import ProgressBar\nfrom ignite.engine import Events\nfrom numpy.random.mtrand import RandomState\nfrom torch.utils.data.dataloader import DataLoader\n\n# --- Training setting ---\n\ntrain_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)\nvalid_loader = DataLoader(valid_dataset, batch_size=batch_size, shuffle=False)\n\noptimizer = torch.optim.Adam(classifier.parameters(), lr=0.001)\nscheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(\n    optimizer, mode='min', factor=0.7, patience=5, min_lr=1e-10)\n\ntrainer = create_trainer(classifier, optimizer, device)\ndef output_transform(output):\n    metric, pred_y, y = output\n    return pred_y.cpu(), y.cpu()\nEpochMetric(\n    compute_fn=macro_recall,\n    output_transform=output_transform\n).attach(trainer, 'recall')\n\npbar = ProgressBar()\npbar.attach(trainer, metric_names='all')\n\nevaluator = create_evaluator(classifier, device)\nEpochMetric(\n    compute_fn=macro_recall,\n    output_transform=output_transform\n).attach(evaluator, 'recall')\n\ndef run_evaluator(engine):\n    evaluator.run(valid_loader)\n\ndef schedule_lr(engine):\n    # metrics = evaluator.state.metrics\n    metrics = engine.state.metrics\n    avg_mae = metrics['loss']\n\n    # --- update lr ---\n    lr = scheduler.optimizer.param_groups[0]['lr']\n    scheduler.step(avg_mae)\n    log_report.report('lr', lr)\n\ntrainer.add_event_handler(Events.EPOCH_COMPLETED, run_evaluator)\ntrainer.add_event_handler(Events.EPOCH_COMPLETED, schedule_lr)\nlog_report = LogReport(evaluator, outdir)\ntrainer.add_event_handler(Events.EPOCH_COMPLETED, log_report)\ntrainer.add_event_handler(\n    Events.EPOCH_COMPLETED,\n    ModelSnapshotHandler(predictor, filepath=outdir / 'predictor.pt'))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"trainer.run(train_loader, max_epochs=10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_history = log_report.get_dataframe()\ntrain_history.to_csv(outdir / 'log.csv', index=False)\n\ntrain_history","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"pred\"></a>\n# Prediction\n\nPlease refer **[Bengali: SEResNeXt prediction with pytorch](https://www.kaggle.com/corochann/bengali-seresnext-prediction-with-pytorch)** for the prediction with trained model and submission for this competition!!!"},{"metadata":{},"cell_type":"markdown","source":"<a id=\"ref\"></a>\n# Reference and further reading\n\n#### Kernel\n\n**[Bangali.AI super fast data loading with feather](https://www.kaggle.com/corochann/bangali-ai-super-fast-data-loading-with-feather)**<br>\nSimple example of how use feather format data to load data faster.\n\n**[Bengali: albumentations data augmentation tutorial](https://www.kaggle.com/corochann/bengali-albumentations-data-augmentation-tutorial)**<br>\nTutorial for Data augmentations with albumentations library.\n\n**[Bengali: SEResNeXt prediction with pytorch](https://www.kaggle.com/corochann/bengali-seresnext-prediction-with-pytorch)**<br>\n**Prediction code of this kernel's trained model, please check this too!**\n\n**[Deep learning - CNN with Chainer: LB 0.99700](https://www.kaggle.com/corochann/deep-learning-cnn-with-chainer-lb-0-99700)**<br>\nData augmentation idea is based on this kernel, which achieves quite high accuracy on MNIST task.\n\n#### Dataset\n**[bengaliai-cv19-feather](https://www.kaggle.com/corochann/bengaliaicv19feather)**<br>\nFeather format dataset\n\n**[bengaliaicv19_seresnext101_32x4d](https://www.kaggle.com/corochann/bengaliaicv19-seresnext101-32x4d)**<br>\n**Trained model weight with this kernel(v1)**\n\n**[bengaliaicv19_trainedmodels](https://www.kaggle.com/corochann/bengaliaicv19-trainedmodels)**<br>\n**Trained model weight with this kernel(v2~)**\n\n#### Library\n**https://github.com/pytorch/ignite**\n\nUsed for training code abstraction. The advantage of abstracting the code is that we can re-use implemented handler class for other training, other competition.<br>\nYou don't need to write code for saving models, logging training loss/metric, show progressbar etc.\n\n**https://github.com/Cadene/pretrained-models.pytorch**\n\nMany pretrained models are supported by this library, and we can switch to use them easily.\nOther model may perform better in this competition.\n\n**https://github.com/albumentations-team/albumentations**\n\nfast image augmentation library and easy to use wrapper around other libraries https://arxiv.org/abs/1809.06839<br>\nI could not show all the methods, you can find more methods in the library, check yourself!"},{"metadata":{},"cell_type":"markdown","source":"<h3 style=\"color:red\">If this kernel helps you, please upvote to keep me motivated :)<br>Thanks!</h3>"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}