{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Chainer Starter Kernel for iMet Collection 2019 - FGVC6\n\n### What is this kernel?\n\nThis is a baseline example using [Chainer](https://docs.chainer.org/en/v5.3.0/) and [ChainerCV](https://chainercv.readthedocs.io/en/stable/).  \nI share this kernel mainly for practice of writing kernel, and for sharing some (maybe useful) information e.g. training settings.\n\n### Summary of model, training, and inference\n#### base model: SEResNet152\n* pre-trained weights on ImageNet\n* obtain output of Global Average Pooling Layer (pool5) and feed it to Dense Layer\n* input shape: (ch, height, witdh) = (3, 128, 128)\n* preprocessing for images\n    * subtract per-channel mean of all train images, and devide by 255 (after data augmentation)\n\n#### training\n* fine-tuning all over the model, not freezing any layer\n* data augmentation\n    * horizontal flip\n    * [random_distort](https://chainercv.readthedocs.io/en/stable/reference/links/ssd.html#random-distort)\n    * random_rotate(angle $\\in$ [-10, 10])\n    * random_expand => [resize_with_random_interpolation](https://chainercv.readthedocs.io/en/stable/reference/links/ssd.html#resize-with-random-interpolation)\n    * random crop\n* max epoch: 20\n* batch size: 128\n* optimizer: NesterovSGD\n    * momentum = 0.9, weight decay = 1e-04\n* learning schedule: cosine anealing\n    * max_lr = 0.01, min_lr = 0.0001\n    * I ran only one cycle.\n    * Note: decaying learning rate by epoch, not iteration (for simple implementation)\n* loss: Focal Loss\n    * alpha = 0.5, gamma = 2\n    * Note:\n        * Loss for each sample is calculated by **summation** of focal loss for each class.\n        * Loss for mini-batch  is calculated by averaging loss for samples in it.\n            * At first I calculated each sample's loss by **averaging**, but it didn't work well.  \n* validation\n    * make one validation set, **not** perform k-fold cross validation\n    * **randomly** split, not considering target(attribute) frequency\n        * train : valid = 4 : 1\n    * check each epoch's f-beta score by threshold = 0.2\n        * since f-beta weights recall higher than precision\n        * [default threshold of fastai's implementation is 0.2](https://github.com/fastai/fastai/blob/master/fastai/metrics.py#L12)\n\n#### inference\n* not using TTA\n* using best threshold for validation set\n    * Thresholds for all classes are same. "},{"metadata":{},"cell_type":"markdown","source":"## setup"},{"metadata":{},"cell_type":"markdown","source":"### import"},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"from time import time\nbeginning_time = time()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import os\nimport gc\nimport json\nimport sys\nimport random\nfrom glob import glob\n\nfrom PIL import Image\nfrom collections import OrderedDict\nfrom joblib import Parallel, delayed\nfrom tqdm._tqdm_notebook import tqdm_notebook\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nfrom math import cos, pi\nfrom sklearn.metrics import precision_recall_curve\n\nimport seaborn as sns\nfrom matplotlib import pyplot as plt\n\ntqdm_notebook.pandas()\n%matplotlib inline\nprint(os.listdir(\"../input\"))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":false},"cell_type":"markdown","source":"#### import Chainer and ChainerCV"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import chainer\nfrom chainer import cuda, functions, links, datasets\nfrom chainer import iterators, optimizers, training, reporter\nfrom chainer import initializers, serializers\n\nimport chainercv\nfrom chainercv import transforms\nfrom chainercv.links.model.ssd import random_distort\nfrom chainercv.links.model.ssd import resize_with_random_interpolation\n\nprint(\"chainercsv:\", chainercv.__version__)\nprint(\"chainer:\", chainer.__version__)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":false},"cell_type":"markdown","source":"#### set data path"},{"metadata":{"trusted":true},"cell_type":"code","source":"DATA_DIR = \"../input/imet-2019-fgvc6\"\nPRETRAINED_MODEL_DIR = \"../input/chainercv-seresnet\"","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print(\"../input\")\nfor path in glob(\"../input/*\"):\n    print(\"\\t|- {}/\".format(path.split(\"/\")[-1]))\n    for fname in os.listdir(path):\n        print(\"\\t\\t|-{}\".format(fname))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### set other imformation."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"img_chmean_train = np.array([164.82181258, 155.93463791, 144.58968491], dtype=\"f\")\n# img_chmean_test = np.array([163.39652723, 154.7340003 , 143.86426686], dtype=\"f\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### settings"},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"seed = 1086\nsettings = OrderedDict(\n    # # model setting.\n    base_model=\"SEResNet152\",\n    n_class=1103,\n    image_size=[128, 128],\n    \n    # # training setting.\n    # valid_fold=0,\n    max_epoch=20,\n    batch_size=128,\n    da_select=[\n        \"random_distort\",\n        \"random_lr_flip\",\n        \"random_rotate\",\n        \"random_expand\", \"resize_with_random\",\n        \"random_crop\"\n    ],\n    learning_schedule=\"cosine\",\n    epoch_per_cycle=20,\n    optimizer=\"NesterovAG\",\n    learning_rate=0.01,\n    learning_rate_min=0.0001,\n    momentum=0.9,\n    weight_decay_rate=1e-04,\n    loss_func=\"FocalLoss\",\n    alpha=0.5,\n    gamma=2,\n)\nsettings[\"pretrained_model_path\"] = \"{}/{}\".format(\n    PRETRAINED_MODEL_DIR,\n    {\n        \"SEResNet50\": \"se_resnet50_imagenet_converted_2018_06_25.npz\",\n        \"SEResNet101\": \"se_resnet101_imagenet_converted_2018_06_25.npz\",\n        \"SEResNet152\": \"se_resnet152_imagenet_converted_2018_06_25.npz\",\n    }[settings[\"base_model\"]])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### classes and functions definition"},{"metadata":{},"cell_type":"markdown","source":"#### model\n* CNN model\n* wapper for training"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"base_class = getattr(chainercv.links, settings[\"base_model\"])\n\nclass FeatureExtractor(base_class):\n    \"\"\"image feture extractor based on pretrained model.\"\"\"\n    \n    def __init__(self, pretrained_model_path, extract_layers=[\"pool5\"]):\n        \"\"\"Initialze.\"\"\"\n        super(FeatureExtractor, self).__init__(pretrained_model=pretrained_model_path)\n        self._pick = extract_layers\n        self.remove_unused()\n    \n    def __call__(self, x):\n        \"\"\"Simply Forward.\"\"\"\n        h = x\n        for name in self.layer_names:\n            h = self[name](h)\n        return h\n    \nclass Ext2Linear(chainer.Chain):\n    \"\"\"Chain to feed output of Extractor to Fully Connect.\"\"\"\n    \n    def __init__(self, n_class, extractor):\n        \"\"\"Initialize.\"\"\"\n        super(Ext2Linear, self).__init__()\n        with self.init_scope():\n            self.extractor = extractor\n            self.fc = links.Linear(\n                None, n_class, initialW=initializers.Normal(scale=0.01))\n\n    def __call__(self, x):\n        \"\"\"Forward.\"\"\"\n        return self.fc(self.extractor(x))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"class MultiLabelClassifier(links.Classifier):\n    \"\"\"Wrapper for multi label classification model.\"\"\"\n    \n    def __init__(self, predictor, lossfun):\n        \"\"\"Initialize\"\"\"\n        super(MultiLabelClassifier, self).__init__(predictor, lossfun)\n        self.compute_accuracy = False\n        self.f_beta = None\n        self.metfun = self._fbeta_score\n        \n    def __call__(self, x, t):\n        \"\"\"Foward. calc loss and evaluation metric.\"\"\"\n        loss = super().__call__(x, t)\n        self.f_beta = None\n        self.f_beta = self.metfun(self.y, t)\n        reporter.report({'f-beta': self.f_beta}, self)\n        \n        return loss\n    \n    def _fbeta_score(self, y_pred, t, beta=2, th=0.2, epsilon=1e-09):\n        \"\"\"\n        calculate f-beta score.\n        \n        calculate f-bata score along **class-axis(axis=1)** and average them along sample-axis.\n        \"\"\"\n        y_prob = functions.sigmoid(y_pred).data\n        t_pred = (y_prob >= th).astype(\"i\")\n        true_pos = (t_pred * t).sum(axis=1)  # tp\n        pred_pos = t_pred.sum(axis=1)  # tp + fp\n        poss_pos = t.sum(axis=1)  # tp + fn\n        precision = true_pos / (pred_pos + epsilon)\n        recall = true_pos / (poss_pos + epsilon)\n        f_beta_each_id = (1 + beta ** 2) * precision * recall / ((beta ** 2) * precision + recall + epsilon)\n        return functions.mean(f_beta_each_id)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### loss\n* define Focal Loss"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"class FocalLoss:\n    \"\"\"\n    Function for Focal loss.\n    \n    calculates focal loss **for each class**, **sum up** them along class-axis, and **average** them along sample-axis.\n    Take data point x and its logit y = model(x),\n    using prob p = (p_0, ..., p_C)^T = sigmoid(y) and label t,\n    focal loss for each class i caluculated by:\n    \n        loss_{i}(p, t) = - \\alpha' + (1 - p'_i) ** \\gamma * ln(p'_i),\n    \n    where\n        \\alpha' = { \\alpha (t_i = 1)\n                  { 1 - \\alpha (t_i = 0)\n         p'_i   = { p_i (t_i = 1)\n                = ( 1 - p_i (t_i = 0)\n    \"\"\"\n\n    def __init__(self, alpha=0.25, gamma=2):\n        \"\"\"Initialize.\"\"\"\n        super(FocalLoss, self).__init__()\n        self.alpha = alpha\n        self.gamma = gamma\n        \n    def __call__(self, y_pred, t, epsilon=1e-31):\n        \"\"\"\n        Forward.\n        \n        p_dash = t * p + (1 - t) * (1 - p) = (1 - t) + (2 * t - 1) * p\n        \"\"\"\n        p_dash = functions.clip(\n            (1 - t) + (2 * t - 1) * functions.sigmoid(y_pred), epsilon, 1 - epsilon)\n        alpha_dash = (1 - t) + (2 * t - 1) * self.alpha\n        # # [y_pred: (bs, n_class), t: (bs: n_class)] => loss_by_sample_x_class: (bs, n_class)\n        loss_by_sample_x_class = - alpha_dash * (1 - p_dash) ** self.gamma * functions.log(p_dash)\n        # # loss_by_sample_x_class: (bs, n_class) => loss_by_sample: (bs, )\n        loss_by_sample = functions.sum(loss_by_sample_x_class, axis=1)\n        # # loss_by_sample: (bs,) => loss: (1, )\n        return functions.mean(loss_by_sample)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### datasets\n* image preprocessing functions \n* data augmentor class"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def resize_pair(pair, size=settings[\"image_size\"]):\n    img, label = pair\n    img = transforms.resize(img, size=size)\n    return (img, label)\n\ndef scale_and_subtract_mean(pair, mean_value=img_chmean_train):\n    img, label = pair\n    img = (img - mean_value[:, None, None]) / 255.\n    return (img, label)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"class DataAugmentor():\n    \"\"\"DataAugmentor for Image Classification.\"\"\"\n\n    def __init__(\n        self, image_size, image_mean,\n        using_methods=[\"random_ud_flip\",\"random_lr_flip\", \"random_90_rotate\", \"random_crop\"]\n    ):\n        \"\"\"Initialize.\"\"\"\n        self.img_size = image_size\n        self.img_mean = image_mean\n        self.img_mean_single = int(image_mean.mean())\n        self.using_methods = using_methods\n        \n        self.func_dict = {\n            \"random_ud_flip\": self._random_ud_flip,\n            \"random_lr_flip\": self._random_lr_flip,\n            \"random_90_rotate\": self._random_90_rotate,\n            \"random_rotate\": self._random_rotate,\n            \"random_expand\": self._random_expand,\n            \"resize_with_random\": self._resize_with_random,\n            \"random_crop\": self._random_crop,\n            \"random_distort\": random_distort,\n        }\n        # # set da func by given order.\n        self.da_funcs = [self.func_dict[um] for um in using_methods]\n        \n    def __call__(self, pair):\n        \"\"\"Forward\"\"\"\n        img_arr, label = pair\n\n        for func in self.da_funcs:\n            img_arr = func(img_arr)\n            \n        return img_arr, label\n\n    def _random_lr_flip(self, img_arr):\n        \"\"\"left-right flipping.\"\"\"\n        if np.random.randint(2):\n            img_arr = img_arr[:, :, ::-1]\n        return img_arr\n\n    def _random_ud_flip(self, img_arr):\n        \"\"\"up-down flipping.\"\"\"\n        if np.random.randint(2):\n            img_arr = img_arr[:, ::-1, :]\n        return img_arr\n    \n    def _random_90_rotate(self, img_arr):\n        \"\"\"90 angle rotation.\"\"\"\n        if np.random.randint(2):\n            img_arr = img_arr.transpose(0, 2, 1)[:, ::-1, :]\n        return img_arr\n    \n    def _random_rotate(self, img_arr, max_angle=10):\n        \"\"\"random degree rotation.\"\"\"\n        angle = np.random.randint(-max_angle, max_angle + 1)\n        if angle == 0:\n            return img_arr\n        return transforms.rotate(img_arr, angle, fill=self.img_mean_single)\n\n    def _random_expand(self, img_arr):\n        \"\"\"random expansion\"\"\"\n        if np.random.randint(2):\n            return img_arr\n        return transforms.random_expand(img_arr, fill=self.img_mean)\n\n    def _resize_with_random(self, img_arr):\n        \"\"\"resize with random interpolation\"\"\"\n        if img_arr.shape[-2:] == self.img_size:\n            return img_arr\n        return resize_with_random_interpolation(img_arr, self.img_size)\n    \n    def _random_crop(self, img_arr, rate=0.5):\n        \"\"\"Random Cropping.\"\"\"\n        crop_size = self.img_size\n        resize_size = tuple(map(lambda x: int(x * 256 / 224), self.img_size))\n\n        if np.random.randint(2):\n            top = np.random.randint(0, resize_size[0] - crop_size[0])\n            botom = top + crop_size[0]\n            left = np.random.randint(0, resize_size[1] - crop_size[1])\n            right = left + crop_size[1]\n            img_arr = transforms.resize(img_arr, size=resize_size)[:, top:botom, left: right]\n\n        return img_arr","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### training\n* trainer extention for cosine anealing"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"class CosineShift(chainer.training.extension.Extension):\n    \"\"\"\n    Cosine Anealing.\n    \n    reference link: https://github.com/takedarts/resnetfamily/blob/master/src/mylib/training/extensions/cosine_shift.py\n    \"\"\"\n    def __init__(self, attr, value, period, period_mult=1, optimizer=None):\n        self._attr = attr\n        self._value = value\n        self._period = period\n        self._period_mult = period_mult\n        self._optimizer = optimizer\n\n        if not hasattr(self._value, '__getitem__'):\n            self._value = (self._value, 0)\n\n    def initialize(self, trainer):\n        self._update_value(trainer)\n\n    def __call__(self, trainer):\n        self._update_value(trainer)\n\n    def _update_value(self, trainer):\n        optimizer = self._optimizer or trainer.updater.get_optimizer('main')\n        epoch = trainer.updater.epoch\n\n        period_range = self._period\n        period_start = 0\n        period_end = period_range\n\n        while period_end <= epoch:\n            period_start = period_end\n            period_range *= self._period_mult\n            period_end += period_range\n\n        n_max, n_min = self._value\n        t_cur = epoch - period_start\n        t_i = period_range\n        value = n_min + 0.5 * (n_max - n_min) * (1 + cos((t_cur / t_i) * pi))\n\n        setattr(optimizer, self._attr, value)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### inference\n* perform predict\n* calculate f-beta score\n* find threshold\n* make pred_ids"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def predict(model, val_iter, gpu_device=-1):\n    val_pred_list = []\n    val_label_list = []\n    iter_num = 0\n    epoch_test_start = time()\n\n    while True:\n        val_batch = val_iter.next()\n        iter_num += 1\n        print(\"\\rtmp_valid_iteration: {:0>5}\".format(iter_num), end=\"\")\n        feature_val, label_val = chainer.dataset.concat_examples(val_batch, gpu_device)\n\n        # Forward the test data\n        with chainer.no_backprop_mode() and chainer.using_config(\"train\", False):\n            prediction_val = model(feature_val)\n            val_pred_list.append(prediction_val)\n            val_label_list.append(label_val)\n            prediction_val.unchain_backward()\n\n        if val_iter.is_new_epoch:\n            print(\" => valid end: {:.2f} sec\".format(time() - epoch_test_start))\n            val_iter.epoch = 0\n            val_iter.current_position = 0\n            val_iter.is_new_epoch = False\n            val_iter._pushed_position = None\n            break\n\n    val_pred_all = cuda.to_cpu(functions.concat(val_pred_list, axis=0).data)\n    val_label_all = cuda.to_cpu(functions.concat(val_label_list, axis=0).data)\n    return val_pred_all, val_label_all","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def average_fbeta_score(y_prob, t, th=0.2, beta=2, epsilon=1e-09):\n    t_pred = (y_prob >= th).astype(int)\n    # # t_pred, t: (sample_num, n_class) => true_pps, predicted_pos, poss_pos : (sample_num,)\n    true_pos = (t_pred * t).sum(axis=1)\n    pred_pos = t_pred.sum(axis=1)\n    poss_pos = t.sum(axis=1)\n\n    p_arr = true_pos / (pred_pos + epsilon)\n    r_arr = true_pos / (poss_pos + epsilon)\n    # # p_arr, r_arr : (n_class,) => f_beta: (n_class)\n    f_beta = (1 + beta ** 2) * p_arr * r_arr / ((beta ** 2) * p_arr + r_arr + epsilon)\n    return f_beta.mean()\n\ndef search_best_threshold(y_prob, t, eval_func, search_range=[0.05, 0.95], interval=0.01):\n    tmp_th = search_range[0]\n    best_th = 0\n    best_eval = -(10**9 + 7)\n    while tmp_th < search_range[1]:\n        eval_score = eval_func(y_prob, t, th=tmp_th)\n        print(tmp_th, eval_score)\n        if eval_score > best_eval:\n            best_th = tmp_th\n            best_eval = eval_score\n        tmp_th += interval\n    return best_th, best_eval\n\ndef make_pred_ids(test_pred, th):\n    class_array = np.arange(test_pred.shape[1])\n    test_cond = test_pred >= th\n    pred_ids = [\" \".join(map(str, class_array[cond])) for cond in test_cond]\n    return pred_ids","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"[end of setup]: {:.3f}\".format(time() - beginning_time))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## prepare data"},{"metadata":{"_kg_hide-input":false},"cell_type":"markdown","source":"### read data"},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"train_df = pd.read_csv(\"{}/train.csv\".format(DATA_DIR))\ntest_df = pd.read_csv(\"{}/sample_submission.csv\".format(DATA_DIR))\nlabels_df = pd.read_csv(\"{}/labels.csv\".format(DATA_DIR))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### make datasets"},{"metadata":{},"cell_type":"markdown","source":"#### label"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ntrain_attr_ohot = np.zeros((len(train_df), len(labels_df)), dtype=\"i\")\nfor idx, attr_arr in enumerate(\n    train_df.attribute_ids.str.split(\" \").apply(lambda l: list(map(int, l))).values):\n    train_attr_ohot[idx, attr_arr] = 1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":false,"_kg_hide-input":false},"cell_type":"code","source":"print(train_attr_ohot.shape)\ntrain_attr_ohot[:5,:20]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### labeled datasets"},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"all_train_dataset = datasets.LabeledImageDataset(\n    pairs=list(zip((train_df.id + \".png\").tolist(), train_attr_ohot)),\n    root=\"{}/train\".format(DATA_DIR))\n\n# # For test set, I set dummy label.\ntest_dataset = datasets.LabeledImageDataset(\n    pairs=list(zip((test_df.id + \".png\").tolist(), [-1] * len(test_df))),\n    root=\"{}/test\".format(DATA_DIR))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### split all train into train and valid\nHere, I split train randomly."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_dataset, valid_dataset = datasets.split_dataset_random(\n    all_train_dataset, first_size=int(len(all_train_dataset) * 0.8), seed=seed)\nprint(\"train set:\", len(train_dataset))\nprint(\"valid set:\", len(valid_dataset))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### set transforms and data augmentations"},{"metadata":{"trusted":true},"cell_type":"code","source":"# # train set, including data augmentation\ntrain_dataset = datasets.TransformDataset(train_dataset, resize_pair)\ntrain_dataset = datasets.TransformDataset(\n    train_dataset, DataAugmentor(\n        image_size=settings[\"image_size\"], image_mean=img_chmean_train,\n        using_methods=settings[\"da_select\"])\n)\ntrain_dataset = datasets.TransformDataset(train_dataset, scale_and_subtract_mean)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# # validt set.\nvalid_dataset = datasets.TransformDataset(valid_dataset, resize_pair)\nvalid_dataset = datasets.TransformDataset(valid_dataset, scale_and_subtract_mean)\n# # test set.\ntest_dataset = datasets.TransformDataset(test_dataset, resize_pair)\ntest_dataset = datasets.TransformDataset(test_dataset, scale_and_subtract_mean)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"[end of preparing data]: {:.3f}\".format(time() - beginning_time))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## make trainer"},{"metadata":{},"cell_type":"markdown","source":"### model"},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"model = Ext2Linear(\n    settings[\"n_class\"], FeatureExtractor(settings[\"pretrained_model_path\"]))\ntrain_model = MultiLabelClassifier(\n    model, lossfun=FocalLoss(settings[\"alpha\"], settings[\"gamma\"]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### optimizer"},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"opt_class = getattr(optimizers, settings[\"optimizer\"])\nif settings[\"optimizer\"] != \"Adam\":\n    optimizer = opt_class(lr=settings[\"learning_rate\"], momentum=settings[\"momentum\"])\n    optimizer.setup(train_model)\n    optimizer.add_hook(chainer.optimizer.WeightDecay(settings[\"weight_decay_rate\"]))\nelse:\n    optmizer = opt_class(\n        alpha=settings[\"learning_rate\"], weight_decay_rate=settings[\"weight_decay_rate\"])\n    optimizer.setup(train_model)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### iterator"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_iter = iterators.MultiprocessIterator(train_dataset, settings[\"batch_size\"])\nvalid_iter = iterators.MultiprocessIterator(\n    valid_dataset, settings[\"batch_size\"], repeat=False, shuffle=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### updater, trainer"},{"metadata":{"trusted":true},"cell_type":"code","source":"updater = training.StandardUpdater(train_iter, optimizer, device=0)\ntrainer = training.trainer.Trainer(updater, stop_trigger=(settings[\"max_epoch\"], \"epoch\"), out=\"training_result\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### trainer extensions"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"logging_attributes = [\"epoch\", \"main/loss\", \"val/main/loss\", \"val/main/f-beta\", \"elapsed_time\", \"lr\"]\n\n# # cosine anealing.\ntrainer.extend(\n    CosineShift('lr', [settings[\"learning_rate\"], settings[\"learning_rate_min\"]], settings[\"epoch_per_cycle\"], 1))\n# # evaluator.\ntrainer.extend(\n    training.extensions.Evaluator(valid_iter, optimizer.target, device=0), name='val',trigger=(1, 'epoch'))\n# # log.\ntrainer.extend(training.extensions.observe_lr(), trigger=(1, 'epoch'))\ntrainer.extend(training.extensions.LogReport(logging_attributes), trigger=(1, 'epoch'))\n# # standard output.\ntrainer.extend(training.extensions.PrintReport(logging_attributes), trigger=(1, 'epoch'))\ntrainer.extend(training.extensions.ProgressBar(update_interval=200))\n# # plots.\ntrainer.extend(\n    training.extensions.PlotReport([\"main/loss\", \"val/main/loss\"], \"epoch\", file_name=\"loss.png\"), trigger=(1, \"epoch\"))\ntrainer.extend(\n    training.extensions.PlotReport([\"val/main/f-beta\"], \"epoch\", file_name=\"fbeta_at02.png\"), trigger=(1, \"epoch\"))\n# # snapshot.\ntrainer.extend(\n    training.extensions.snapshot(filename='snapshot_epoch_{.updater.epoch}.npz'), trigger=(10, 'epoch'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"[end of preparing trainer]: {:.3f}\".format(time() - beginning_time))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## training"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ntrainer.run()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# # save last model\ntrained_model = trainer.updater.get_optimizer('main').target.predictor\nserializers.save_npz('{}/epoch{:0>3}.model'.format(\"training_result\", settings[\"max_epoch\"]), trained_model)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## inference"},{"metadata":{},"cell_type":"markdown","source":"### find best thr for valid set"},{"metadata":{"trusted":true},"cell_type":"code","source":"valid_iter = iterators.MultiprocessIterator(\n    valid_dataset, settings[\"batch_size\"], repeat=False, shuffle=False)\nval_pred, val_label = predict(trained_model, valid_iter, gpu_device=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"val_prob = functions.sigmoid(val_pred).data\nbest_th, best_fbeta = search_best_threshold(\n    val_prob, val_label, eval_func=average_fbeta_score, search_range=[0.1, 0.9], interval=0.01)\nprint(best_th, best_fbeta)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### submit"},{"metadata":{"trusted":true},"cell_type":"code","source":"test_iter = iterators.MultiprocessIterator(\n    test_dataset, settings[\"batch_size\"], repeat=False, shuffle=False)\ntest_pred, _ = predict(trained_model, test_iter, gpu_device=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_prob = functions.sigmoid(test_pred).data\ntest_pred_ids = make_pred_ids(test_prob, th=best_th)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df.attribute_ids = test_pred_ids\nprint(test_df.shape)\ntest_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df.to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}