{"cells":[{"metadata":{"_uuid":"eb587e5d120c6efc3c0a46a12b43bfd6cbee990c"},"cell_type":"markdown","source":"### General information\n\nThere are many frameworks for Deep Learning. Recently I started using Pytorch and liked it, but sometimes it requires too much coding, so I started looking for wrappers over Pytorch.\n\nRecently a new library emerged: Kekas https://github.com/belskikh/kekas\n\nIt provides a simple API for training neural nets for images and good visualizations of training process. There are also nice tricks from fast.ai - learning rate finder, one cycle policy and others.\n\nIn this kernel I'll show what this library can do. At first I'll take a small subset of data and demonstrate the general functionality, after this we'll try training model on the full data.  Sadly, kaggle kernels have some problems currently, so I won't be able to show everything.\n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# libraries\nimport numpy as np\nimport pandas as pd\nimport os\nimport cv2\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import roc_auc_score\nimport torch\nfrom torch.utils.data import TensorDataset, DataLoader,Dataset\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torchvision\nimport torchvision.transforms as transforms\nimport torch.optim as optim\nfrom torch.optim import lr_scheduler\nimport time \nfrom PIL import Image\ntrain_on_gpu = True\nfrom torch.utils.data.sampler import SubsetRandomSampler\nfrom torch.optim.lr_scheduler import StepLR, ReduceLROnPlateau, CosineAnnealingLR\nfrom sklearn.metrics import accuracy_score\nimport cv2","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fb541b2ee1cf77760101734c0dbacd064e205263"},"cell_type":"markdown","source":"Some of good libraries for DL aren't available in Docker with GPU by default, so it is necessary to install them. (don't forget to turn on internet connection in kernels)."},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"!pip install albumentations > /dev/null 2>&1\n!pip install pretrainedmodels > /dev/null 2>&1\n!pip install kekas > /dev/null 2>&1\n!pip install adabound > /dev/null 2>&1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"80f4c3aeb8c0d777aa3315343f2e14430c1eb744","_kg_hide-input":true},"cell_type":"code","source":"# more imports\nimport albumentations\nfrom albumentations import torch as AT\nimport pretrainedmodels\nimport adabound\n\nfrom kekas import Keker, DataOwner, DataKek\nfrom kekas.transformations import Transformer, to_torch, normalize\nfrom kekas.metrics import accuracy\nfrom kekas.modules import Flatten, AdaptiveConcatPool2d\nfrom kekas.callbacks import Callback, Callbacks, DebuggerCallback\nfrom kekas.utils import DotDict","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0a8348ba0c4235b0791b9a92f79f076beee54a56"},"cell_type":"markdown","source":"## Preparing data"},{"metadata":{"trusted":true,"_uuid":"96108ea990382ebc33fe578725e6eb44bb7c8e35"},"cell_type":"code","source":"labels = pd.read_csv('../input/train_labels.csv')\nfig = plt.figure(figsize=(25, 4))\n# display 20 images\ntrain_imgs = os.listdir(\"../input/train\")\nfor idx, img in enumerate(np.random.choice(train_imgs, 20)):\n    ax = fig.add_subplot(2, 20//2, idx+1, xticks=[], yticks=[])\n    im = Image.open(\"../input/train/\" + img)\n    plt.imshow(im)\n    lab = labels.loc[labels['id'] == img.split('.')[0], 'label'].values[0]\n    ax.set_title(f'Label: {lab}')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bc472281b89644f0dd471c5d24e139eec04bc80d"},"cell_type":"markdown","source":"Kekas accepts pandas DataFrame as an input and iterates over it to get image names and labels"},{"metadata":{"trusted":true,"_uuid":"dc72ee92ea4aa48fd248cee2033d009fbf233fee"},"cell_type":"code","source":"test_img = os.listdir('../input/test/')\ntest_df = pd.DataFrame(test_img, columns=['id'])\ntest_df['label'] = -1\ntest_df['data_type'] = 'test'\ntest_df['id'] = test_df['id'].apply(lambda x: x.split('.')[0])\n\nlabels['label'] = labels['label'].astype(int)\nlabels['data_type'] = 'train'\n\nlabels.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2366c4a60f771782946fe628715c605f2c14a3e9"},"cell_type":"code","source":"labels.shape, test_df.shape","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"61f6b0bbd68f0789842c10f7b0e61fde7943f67d"},"cell_type":"markdown","source":"For now let's take a subset of data for faster training."},{"metadata":{"trusted":true,"_uuid":"fd0f13b7fdd56023a80f4933e9a3c9c509ad9a19"},"cell_type":"code","source":"train_short, _ = train_test_split(labels, stratify=labels.label, test_size=0.95)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2da21764c1c6683756e8791bdb0d4b5cbae87779"},"cell_type":"code","source":"train_short.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f985cdaf0d2828fe2af0594125f92ad289b373dd"},"cell_type":"code","source":"# splitting data into train and validation\ntrain, valid = train_test_split(train_short, stratify=train_short.label, test_size=0.1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"01e4caf07f569e26d77a18b2164279d3f8251d93"},"cell_type":"markdown","source":"### Reader function\n\nAt first it is necessary to create a reader function, which will open images. It accepts i and row as input (like from pandas iterrows). The function should return a dictionary with image and label.\n\n`[:,:,::-1]` - is a neat trick which converts BGR images to RGB, it works faster that converting to RGB by usual means."},{"metadata":{"trusted":true,"_uuid":"71a3b7c0b26542921447fc68b7027b5b03f1279e"},"cell_type":"code","source":"def reader_fn(i, row):\n    image = cv2.imread(f\"../input/{row['data_type']}/{row['id']}.tif\")[:,:,::-1] # BGR -> RGB\n    label = torch.Tensor([row[\"label\"]])\n    return {\"image\": image, \"label\": label}","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"404e56b9037e3e05c8618c079307a0f966145495"},"cell_type":"markdown","source":"### Data transformation\n\nNext step is defining data transformations and augmentations. This differs from standard PyTorch way. We define resizing, augmentations and normalizing separately, this allows to easily create separate transformers for train and valid/test data.\n\nAt first we define augmentations. We create a function with a list of augmentations (I prefer albumentation library: https://github.com/albu/albumentations)"},{"metadata":{"trusted":true,"_uuid":"e87c7bc77e94d12a76207467c214f10c01a72d9f"},"cell_type":"code","source":"def augs(p=0.5):\n    return albumentations.Compose([\n        albumentations.HorizontalFlip(),\n        albumentations.RandomRotate90(),\n        albumentations.Transpose(),\n        albumentations.Flip(),\n    ], p=p)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"55933138a1b6c70fb9efae9810efde81af17bcb2"},"cell_type":"markdown","source":"Now we create a transforming function. It heavily uses Transformer from kekas.\n* The first step is defining resizing. You can change arguments of function if you want images to have different height and width, otherwis you can leave it as it is.\n* Next step is defining atgmentations. Here we provide the key of image which is defined in `reader_fn`;\n* The third step is defining final transformation to tensor and normalizing;\n* After this we can compose separate transformations for train and valid/test data;"},{"metadata":{"trusted":true,"_uuid":"c5e75929841ba4d303d689aa646ac909361c5d30"},"cell_type":"code","source":"def get_transforms(dataset_key, size, p):\n\n    PRE_TFMS = Transformer(dataset_key, lambda x: cv2.resize(x, (size, size)))\n\n    AUGS = Transformer(dataset_key, lambda x: augs()(image=x)[\"image\"])\n\n    NRM_TFMS = transforms.Compose([\n        Transformer(dataset_key, to_torch()),\n        Transformer(dataset_key, normalize())\n    ])\n    \n    train_tfms = transforms.Compose([PRE_TFMS, AUGS, NRM_TFMS])\n    val_tfms = transforms.Compose([PRE_TFMS, NRM_TFMS])\n    \n    return train_tfms, val_tfms","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"964691645ae2503ce9ace132871845f91368c388"},"cell_type":"code","source":"# getting transformations\ntrain_tfms, val_tfms = get_transforms(\"image\", 224, 0.5)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ec628937c544201e1d0e58a4eb7693180aef62b4"},"cell_type":"markdown","source":"Now we can create a DataKek, which is similar to creating dataset in Pytorch. We define the data, reader function and transformation.Then we can define standard PyTorch DataLoader."},{"metadata":{"trusted":true,"_uuid":"2dbf5eb97b30b32d04620d14ad9c0f0d57168f1e"},"cell_type":"code","source":"train_dk = DataKek(df=train, reader_fn=reader_fn, transforms=train_tfms)\nval_dk = DataKek(df=valid, reader_fn=reader_fn, transforms=val_tfms)\n\nbatch_size = 32\nworkers = 0\n\ntrain_dl = DataLoader(train_dk, batch_size=batch_size, num_workers=workers, shuffle=True, drop_last=True)\nval_dl = DataLoader(val_dk, batch_size=batch_size, num_workers=workers, shuffle=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f7ad46e37ad5adab5a160bc5301c1b2114237fbb"},"cell_type":"code","source":"test_dk = DataKek(df=test_df, reader_fn=reader_fn, transforms=val_tfms)\ntest_dl = DataLoader(test_dk, batch_size=batch_size, num_workers=workers, shuffle=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5f5cdb09e7f69d31688361466e79c5240e13898e"},"cell_type":"markdown","source":"## Building a neural net\n\nHere we define the architecture of the neural net.\n\n* Pre-trained backbone is taken from pretrainedmodels: https://github.com/Cadene/pretrained-models.pytorch Here I take `densenet169`\n* We also define changes to the architecture. For example, we take off the last layer and add a custom head with `nn.Sequential`. `AdaptiveConcatPool2d` is a layer in kekas, which concats `AdaptiveMaxPooling` and `AdaptiveAveragePooling` "},{"metadata":{"trusted":true,"_uuid":"9b4125a576d202aeb141b9634eb2519e598f221a"},"cell_type":"code","source":"# create a simple neural network using pretrainedmodels library\n# https://github.com/Cadene/pretrained-models.pytorch\n\nclass Net(nn.Module):\n    def __init__(\n            self,\n            num_classes: int,\n            p: float = 0.2,\n            pooling_size: int = 2,\n            last_conv_size: int = 81536,\n            arch: str = \"densenet169\",\n            pretrained: str = \"imagenet\") -> None:\n        \"\"\"A simple model to finetune.\n        \n        Args:\n            num_classes: the number of target classes, the size of the last layer's output\n            p: dropout probability\n            pooling_size: the size of the result feature map after adaptive pooling layer\n            last_conv_size: size of the flatten last backbone conv layer\n            arch: the name of the architecture form pretrainedmodels\n            pretrained: the mode for pretrained model from pretrainedmodels\n        \"\"\"\n        super().__init__()\n        net = pretrainedmodels.__dict__[arch](pretrained=pretrained)\n        modules = list(net.children())[:-1]  # delete last layer\n        # add custom head\n        modules += [nn.Sequential(\n            # AdaptiveConcatPool2d is a concat of AdaptiveMaxPooling and AdaptiveAveragePooling \n            AdaptiveConcatPool2d(size=pooling_size),\n            Flatten(),\n            nn.BatchNorm1d(13312),\n            nn.Dropout(p),\n            nn.Linear(13312, num_classes)\n        )]\n        self.net = nn.Sequential(*modules)\n\n    def forward(self, x):\n        logits = self.net(x)\n        return logits","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f17467fad031c647f943164186dfc28459873bdb"},"cell_type":"markdown","source":"The data for training needs to be transformed one more time - we define DataOwner, which contains all the data. For now let's define it for train and valid."},{"metadata":{"trusted":true,"_uuid":"f7bf1eda9752d67d3223926968dde5ae9b890b95"},"cell_type":"code","source":"dataowner = DataOwner(train_dl, val_dl, None)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ae5e1a8f6320f39ab0710f20e7817ed158f66664"},"cell_type":"markdown","source":"Next we define model and loss. As I choose `BCEWithLogitsLoss`, we can set the number of classes for output to 1."},{"metadata":{"trusted":true,"_uuid":"36624dd0f75523211887da2d9ab48efbd4e39159"},"cell_type":"code","source":"model = Net(num_classes=1)\n\ncriterion = nn.BCEWithLogitsLoss()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fc11e642ce985b7501af080738dbdb86564c7a39"},"cell_type":"markdown","source":"And now we define what will the model do with the data. For example we could slice the output and take only a part of it. For now we will simply return the output of the model."},{"metadata":{"trusted":true,"_uuid":"f92f0e7f2a7f0ed976d058a6e7ed1bcad986d083"},"cell_type":"code","source":"def step_fn(model: torch.nn.Module,\n            batch: torch.Tensor) -> torch.Tensor:\n    \"\"\"Determine what your model will do with your data.\n\n    Args:\n        model: the pytorch module to pass input in\n        batch: the batch of data from the DataLoader\n\n    Returns:\n        The models forward pass results\n    \"\"\"\n    \n    inp = batch[\"image\"]\n    return model(inp)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"16b4865d41af3f8092c5151b2e8c8d5146fa76d0"},"cell_type":"markdown","source":"### Defining custom metrics\n\nCurrently Kekas has only simple accuracy (see here: https://github.com/belskikh/kekas/blob/9f9fbb00ded7e12e79ba3d2edaa3a8f82b368722/kekas/metrics.py), but we can define any metric we like. It should accept tensors of target and prediction as an imput and return the metric value as an output.\n\nLet's define an accuracy for out net with BCE."},{"metadata":{"trusted":true,"_uuid":"1b886dabc4209e2643f5285b601114736ded97af"},"cell_type":"code","source":"def bce_accuracy(target: torch.Tensor,\n                 preds: torch.Tensor) -> float:\n    thresh = 0.5\n    target = target.cpu().detach().numpy()\n    preds = (torch.sigmoid(preds).cpu().detach().numpy() > thresh).astype(int)\n    return accuracy_score(target, preds)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1501aeed6180ae2e06edb12c9d3d27ffc6fb0a94"},"cell_type":"markdown","source":"## Keker\n\nNow we can define the Keker - the core Kekas class for training the model.\n\nHere we define everything which is necessary for training:\n* the model which was defined earlier;\n* dataowner containing the data for training and validation;\n* criterion;\n* step function;\n* the key of labels, which was defined in the reader function;\n* the dictionary with metrics (there can be several of them);\n* The optimizer and its parameters;\n\nThere are more advanced parameters, you can read more about them in the docstring: https://github.com/belskikh/kekas/blob/9f9fbb00ded7e12e79ba3d2edaa3a8f82b368722/kekas/keker.py"},{"metadata":{"trusted":true,"_uuid":"9d19c8f39cff40df5b025e3b9ecc62d84eb912af"},"cell_type":"code","source":"keker = Keker(model=model,\n              dataowner=dataowner,\n              criterion=criterion,\n              step_fn=step_fn,\n              target_key=\"label\",\n              metrics={\"acc\": bce_accuracy},\n              opt=torch.optim.SGD,\n              opt_params={\"momentum\": 0.99})","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"32b1da31a8826cc9e9cd43444c8c35a6cff653c5"},"cell_type":"markdown","source":"### Freezing and unfreezing\nFreezing and unfreezing layers is quite important if you want to achieve high score. It can be done easily with Kekas. Af first you unfreeze the model and then freeze up to a necessary layer."},{"metadata":{"trusted":true,"_uuid":"512dd3a705a5957bd7068370ade18aa20a99df97"},"cell_type":"code","source":"keker.unfreeze(model_attr=\"net\")\n\n# If you want to freeze till some layer:\nlayer_num = -1\nkeker.freeze_to(layer_num, model_attr=\"net\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"15649541a1a55d3f9294d30a68d27fd582bfb2f2"},"cell_type":"markdown","source":"## Various ways to train a model"},{"metadata":{"_uuid":"1b81f206302b173d0fbf9e2936ded7668a07e6dd"},"cell_type":"markdown","source":"### Learning Rate Find\nIn fast.ai Jeremy offered and idea of learning rate finder to find an optimal learning rate. Let's do it! We can simply define the final LR and the directory where logs are written."},{"metadata":{"trusted":true,"_uuid":"20375f9fed0e3c0dc972a7d0d2637611933fa99c"},"cell_type":"code","source":"keker.kek_lr(final_lr=0.001, logdir=\"logs\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4d66e37c67dda38d4a25208fe44d718f05b5366e"},"cell_type":"markdown","source":"One of the ways to plot the training curves is to use tensorboard. Sadly it isn't supported well by kaggle. If you are in edit mode, you can try the next cell. Or you can simply use the keker method to plot the logs from the defined directory."},{"metadata":{"trusted":true,"_uuid":"c41de3c0128f0112d77414fdfe9d466c6e730a63"},"cell_type":"code","source":"# !wget https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip\n# !unzip ngrok-stable-linux-amd64.zip\n# LOG_DIR = 'logs' # Here you have to put your log directory\n# get_ipython().system_raw(\n#     'tensorboard --logdir {} --host 0.0.0.0 --port 6006 &'\n#     .format(LOG_DIR)\n# )\n# get_ipython().system_raw('./ngrok http 6006 &')\n# ! curl -s http://localhost:4040/api/tunnels | python3 -c \"import sys, json; print(json.load(sys.stdin)['tunnels'][0]['public_url'])\"\n#!tensorboard --logdir logs --host 0.0.0.0 --port 6007","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2db93e21b9ff744c19c7d451cf8807ea874b1efd"},"cell_type":"code","source":"keker.plot_kek_lr('logs')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6aa652f2544cd850d4dc01d0518b1745a1239bbd"},"cell_type":"markdown","source":"### Training the model"},{"metadata":{"trusted":true,"_uuid":"dfc51984c74214e49f80a372e541e612fe522778"},"cell_type":"code","source":"# simply training the model with pre-defined parameters.\nkeker.kek(lr=2e-5, epochs=1, logdir='train_logs0')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4fe8616e0c993a9ca066155b04901957c37c6d73"},"cell_type":"code","source":"# we plot all three graphs by default, but we can also choose which of them will be shown.\nkeker.plot_kek('train_logs0', metrics=['loss', 'acc'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6120a12953257313fe0f82a9152aeb095f36a9e7"},"cell_type":"markdown","source":"Also we can define new parameters or even override old ones."},{"metadata":{"trusted":true,"_uuid":"ef3c20f96908d14944cb98eeeb6568c5b2475709"},"cell_type":"code","source":"keker.kek(lr=2e-5, epochs=3,\n         sched=torch.optim.lr_scheduler.StepLR,\n              sched_params={\"step_size\":1, \"gamma\": 0.9},# optimizer class. if note specifiyng, \n                                                  # an SGD is using by default\n              opt_params={\"weight_decay\": 1e-5},\n              early_stop_params={\n              \"patience\": 2,   \n              \"metric\": \"acc\", \n              \"mode\": \"min\",   \n              \"min_delta\": 0\n          }, logdir='train_logs')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"da0ffd3be49c2377dbfc2da58d6232704e30d650"},"cell_type":"code","source":"keker.plot_kek('train_logs')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8fe7573f8b69e5e8e5acc3a19da5057903bd3c7a"},"cell_type":"markdown","source":"One more cool feature is a possibility to use one cycle!"},{"metadata":{"trusted":true,"_uuid":"5c77e96f8489c818cc861e62f4663f6a06ff5931"},"cell_type":"code","source":"keker.kek_one_cycle(max_lr=1e-4,                  # the maximum learning rate\n                    cycle_len=3,                  # number of epochs, actually, but not exactly\n                    momentum_range=(0.95, 0.85),  # range of momentum changes\n                    div_factor=25,                # max_lr / min_lr\n                    increase_fraction=0.3,        # the part of cycle when learning rate increases\n                    logdir='train_logs1')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"826403c7dfc35f365730d081a0fd685e08a0ad73"},"cell_type":"code","source":"keker.plot_kek('train_logs1')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b8ca789693963ab2a6a17f0d98ba71309ecf5bd6"},"cell_type":"markdown","source":"There are much more features, you can read the tutorial: https://github.com/belskikh/kekas/blob/master/Tutorial.ipynb or the code itself"},{"metadata":{"_uuid":"d676b9842b2d9114129341240de46623a74b42eb"},"cell_type":"markdown","source":"## Training model on all data\n\nNow, let's train model on all data.\n\nI use AdaBound: https://github.com/Luolc/AdaBound"},{"metadata":{"trusted":true,"_uuid":"fbb1eb561c7e1d255863e7e206dd1add5c06b54c"},"cell_type":"code","source":"train, valid = train_test_split(labels, stratify=labels.label, test_size=0.1)\n\ntrain_dk = DataKek(df=train, reader_fn=reader_fn, transforms=train_tfms)\nval_dk = DataKek(df=valid, reader_fn=reader_fn, transforms=val_tfms)\n\ntrain_dl = DataLoader(train_dk, batch_size=batch_size, num_workers=workers, shuffle=True, drop_last=True)\nval_dl = DataLoader(val_dk, batch_size=batch_size, num_workers=workers, shuffle=False)\n\ntest_dk = DataKek(df=test_df, reader_fn=reader_fn, transforms=val_tfms)\ntest_dl = DataLoader(test_dk, batch_size=batch_size, num_workers=workers, shuffle=False)\n\ndataowner = DataOwner(train_dl, val_dl, None)\n\nkeker = Keker(model=model,\n              dataowner=dataowner,\n              criterion=criterion,\n              step_fn=step_fn,\n              target_key=\"label\",\n              metrics={\"acc\": bce_accuracy},\n              opt=adabound.AdaBound,\n              opt_params={'final_lr': 0.001})\n\nkeker.unfreeze(model_attr=\"net\")\n\nlayer_num = -3\nkeker.freeze_to(layer_num, model_attr=\"net\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"32fae6bf3de3cdd5fdf7f4c62edeb054d4d4a843"},"cell_type":"code","source":"keker.kek_one_cycle(max_lr=1e-4,\n                    cycle_len=6,\n                    momentum_range=(0.95, 0.85),\n                    div_factor=25,\n                    increase_fraction=0.3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a01a043273ae2cb85b92ae1eaf2f02ca7f374563","_kg_hide-input":true},"cell_type":"code","source":"# keker.kek(lr=2e-5, epochs=3,\n#          sched=ReduceLROnPlateau,\n#               sched_params={\"patience\":2, \"factor\": 0.1},# optimizer class. if note specifiyng, \n#                                                   # an SGD is using by default\n#               early_stop_params={\n#               \"patience\": 2,   \n#               \"metric\": \"acc\", \n#               \"mode\": \"min\",   \n#               \"min_delta\": 0\n#           }, logdir='train_logs_fin')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e48efb13bf9bd8ace60e1add4e2fe23b7eff755f"},"cell_type":"code","source":"keker.plot_kek_lr('train_logs_fin')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a4b620668a36bb975ffd2291ec89766c9f73eb42"},"cell_type":"markdown","source":"## Predicting and TTA\n\nSimply predicting on test data is okay, but it is better to use TTA - test time augmentation. Let's see how it can be done with Kekas.\n\n* define augmentations;\n* define augmentation function;\n* create objects with these augmentations;\n* put these objects into a single dictionary;"},{"metadata":{"trusted":true,"_uuid":"182facb711bfb16e86a75c276fad682433f5d08e"},"cell_type":"code","source":"# simple prediction\npreds = keker.predict_loader(loader=test_dl)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"f04b3e274fcd61d5546089740e7b5ac7230fb446"},"cell_type":"code","source":"# TTA\nflip_ = albumentations.Flip(always_apply=True)\ntranspose_ = albumentations.Transpose(always_apply=True)\n\ndef insert_aug(aug, dataset_key=\"image\", size=224):    \n    PRE_TFMS = Transformer(dataset_key, lambda x: cv2.resize(x, (size, size)))\n    \n    AUGS = Transformer(dataset_key, lambda x: aug(image=x)[\"image\"])\n    \n    NRM_TFMS = transforms.Compose([\n        Transformer(dataset_key, to_torch()),\n        Transformer(dataset_key, normalize())\n    ])\n    \n    tfm = transforms.Compose([PRE_TFMS, AUGS, NRM_TFMS])\n    return tfm\n\nflip = insert_aug(flip_)\ntranspose = insert_aug(transpose_)\n\ntta_tfms = {\"flip\": flip, \"transpose\": transpose}\n\n# third, run TTA\nkeker.TTA(loader=test_dl,                # loader to predict on \n          tfms=tta_tfms,                # list or dict of always applying transforms\n          savedir=\"tta_preds1\",  # savedir\n          prefix=\"preds\")               # (optional) name prefix. default is 'preds'\n\n# it will saves predicts for each augmentation to savedir with name\n#  - {prefix}_{name_from_dict}.npy if tfms is a dict\n#  - {prefix}_{index}.npy          if tfms is a list","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"60c0886d9958a50db136d8276c8658a965c8f7c4"},"cell_type":"code","source":"prediction = np.zeros((test_df.shape[0], 1))\nfor i in os.listdir('tta_preds1'):\n    pr = np.load('tta_preds1/' + i)\n    prediction += pr\nprediction = prediction / len(os.listdir('tta_preds1'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"42df31c51c9bdc8d7f70171f2e28753f95f16d81"},"cell_type":"code","source":"test_preds = pd.DataFrame({'imgs': test_df.id.values, 'preds': prediction.reshape(-1,)})\ntest_preds.columns = ['id', 'label']\ntest_preds.to_csv('sub.csv', index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}