{"cells":[{"metadata":{"_kg_hide-input":false,"_kg_hide-output":false},"cell_type":"markdown","source":"# Description\nGreetings, this kernel provides fast.ai starter code for Bengali.AI Handwritten Grapheme Classification competition.\n\nThe task proposed in this competition is recognition of handwritten Bengali letters. In contrast to similar competitions such as [mnist digit recognition](https://www.kaggle.com/c/digit-recognizer) or the recent [Kannada MNIST](https://www.kaggle.com/c/Kannada-MNIST), Bengali alphabet is quite complex and may include ~13,000 different grapheme variations. Fortunately, each grapheme can be decomposed into 3 parts: grapheme_root, vowel_diacritic, and consonant_diacritic (168, 11, and 7 independent classes, respectively). Therefore, the task of grapheme recognition is significantly simplified in comparison with 13k-way classification. Though, additional consideration may be required for this multitask classification, like checking if 3 independent models, or a single model one head, or a single model with 3 heads (this kernel) works the best. This kernel is mostly based on [my findings](https://www.kaggle.com/c/Kannada-MNIST/discussion/122430) in a recent Kannada MNIST toy competition.\n\nIn addition to this kernel I have prepared a [code](https://www.kaggle.com/iafoss/image-preprocessing-128x128) for data preprocessing and generation of cropped images rescaled to 128x128. Because the data has much higher resolution than MNIST or CIFAR, the things that work there may be less applicable to this data, while ImageNet based solutions should be more suitable. So, I've choosen DenseNet121 as a starter network. In addition to this kernel I will prepare a submission kernel posted separately."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"%reload_ext autoreload\n%autoreload 2\n%matplotlib inline\n\nimport fastai\nfrom fastai.vision import *\nfrom fastai.callbacks import SaveModelCallback\nimport os\nfrom sklearn.model_selection import KFold\nfrom radam import *\nfrom csvlogger import *\nfrom mish_activation import *\nimport warnings\nwarnings.filterwarnings(\"ignore\")\n\nfastai.__version__","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sz = 128\nbs = 128\nnfolds = 4 #keep the same split as the initial dataset\nfold = 0\nSEED = 2019\nTRAIN = '../input/grapheme-imgs-128x128/'\nLABELS = '../input/bengaliai-cv19/train.csv'\narch = models.densenet121\n\ndef seed_everything(seed):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.backends.cudnn.deterministic = True\n    torch.backends.cudnn.benchmark = True\n\nseed_everything(SEED)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"df = pd.read_csv(LABELS)\nnunique = list(df.nunique())[1:-1]\nprint(nunique)\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"stats = ([0.0692], [0.2051])\ndata = (ImageList.from_df(df, path='.', folder=TRAIN, suffix='.png', \n        cols='image_id', convert_mode='L')\n        .split_by_idx(range(fold*len(df)//nfolds,(fold+1)*len(df)//nfolds))\n        .label_from_df(cols=['grapheme_root','vowel_diacritic','consonant_diacritic'])\n        .transform(get_transforms(do_flip=False,max_warp=0.1), size=sz, padding_mode='zeros')\n        .databunch(bs=bs)).normalize(stats)\n\ndata.show_batch()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model"},{"metadata":{},"cell_type":"markdown","source":"The starter model is based on DenseNet121, which I found to work quite well for such kind of problems. The first conv is replaced to accommodate for 1 channel input, and the corresponding pretrained weights are summed. ReLU activation in the head is replaced by Mish, which works noticeably better for all tasks I checked it so far. Since each portion of the prediction (grapheme_root, vowel_diacritic, and consonant_diacritic) is quite independent concept, I create a separate head for each of them (though I didn't do a comparison with one head solution)."},{"metadata":{"trusted":true},"cell_type":"code","source":"class Head(nn.Module):\n    def __init__(self, nc, n, ps=0.5):\n        super().__init__()\n        layers = [AdaptiveConcatPool2d(), Mish(), Flatten()] + \\\n            bn_drop_lin(nc*2, 512, True, ps, Mish()) + \\\n            bn_drop_lin(512, n, True, ps)\n        self.fc = nn.Sequential(*layers)\n        self._init_weight()\n        \n    def _init_weight(self):\n        for m in self.modules():\n            if isinstance(m, nn.Conv2d):\n                torch.nn.init.kaiming_normal_(m.weight)\n            elif isinstance(m, nn.BatchNorm2d):\n                m.weight.data.fill_(1.0)\n                m.bias.data.zero_()\n        \n    def forward(self, x):\n        return self.fc(x)\n\n#change the first conv to accept 1 chanel input\nclass Dnet_1ch(nn.Module):\n    def __init__(self, arch=arch, n=nunique, pre=True, ps=0.5):\n        super().__init__()\n        m = arch(True) if pre else arch()\n        \n        conv = nn.Conv2d(1, 64, kernel_size=7, stride=2, padding=3, bias=False)\n        w = (m.features.conv0.weight.sum(1)).unsqueeze(1)\n        conv.weight = nn.Parameter(w)\n        \n        self.layer0 = nn.Sequential(conv, m.features.norm0, nn.ReLU(inplace=True))\n        self.layer1 = nn.Sequential(\n            nn.MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False),\n            m.features.denseblock1)\n        self.layer2 = nn.Sequential(m.features.transition1,m.features.denseblock2)\n        self.layer3 = nn.Sequential(m.features.transition2,m.features.denseblock3)\n        self.layer4 = nn.Sequential(m.features.transition3,m.features.denseblock4,\n                                    m.features.norm5)\n        \n        nc = self.layer4[-1].weight.shape[0]\n        self.head1 = Head(nc,n[0])\n        self.head2 = Head(nc,n[1])\n        self.head3 = Head(nc,n[2])\n        #to_Mish(self.layer0), to_Mish(self.layer1), to_Mish(self.layer2)\n        #to_Mish(self.layer3), to_Mish(self.layer4)\n        \n    def forward(self, x):    \n        x = self.layer0(x)\n        x = self.layer1(x)\n        x = self.layer2(x)\n        x = self.layer3(x)\n        x = self.layer4(x)\n        \n        x1 = self.head1(x)\n        x2 = self.head2(x)\n        x3 = self.head3(x)\n        \n        return x1,x2,x3","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Loss"},{"metadata":{},"cell_type":"markdown","source":"Cross entropy loss is applied independently to each part of the prediction and the result is summed with the corresponding weight."},{"metadata":{"trusted":true},"cell_type":"code","source":"class Loss_combine(nn.Module):\n    def __init__(self):\n        super().__init__()\n        \n    def forward(self, input, target,reduction='mean'):\n        x1,x2,x3 = input\n        x1,x2,x3 = x1.float(),x2.float(),x3.float()\n        y = target.long()\n        return 0.7*F.cross_entropy(x1,y[:,0],reduction=reduction) + 0.1*F.cross_entropy(x2,y[:,1],reduction=reduction) + \\\n          0.2*F.cross_entropy(x3,y[:,2],reduction=reduction)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The code below computes the competition metric and recall macro metrics for individual components of the prediction. The code is partially borrowed from fast.ai."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"class Metric_idx(Callback):\n    def __init__(self, idx, average='macro'):\n        super().__init__()\n        self.idx = idx\n        self.n_classes = 0\n        self.average = average\n        self.cm = None\n        self.eps = 1e-9\n        \n    def on_epoch_begin(self, **kwargs):\n        self.tp = 0\n        self.fp = 0\n        self.cm = None\n    \n    def on_batch_end(self, last_output:Tensor, last_target:Tensor, **kwargs):\n        last_output = last_output[self.idx]\n        last_target = last_target[:,self.idx]\n        preds = last_output.argmax(-1).view(-1).cpu()\n        targs = last_target.long().cpu()\n        \n        if self.n_classes == 0:\n            self.n_classes = last_output.shape[-1]\n            self.x = torch.arange(0, self.n_classes)\n        cm = ((preds==self.x[:, None]) & (targs==self.x[:, None, None])) \\\n          .sum(dim=2, dtype=torch.float32)\n        if self.cm is None: self.cm =  cm\n        else:               self.cm += cm\n\n    def _weights(self, avg:str):\n        if self.n_classes != 2 and avg == \"binary\":\n            avg = self.average = \"macro\"\n            warn(\"average=`binary` was selected for a non binary case. \\\n                 Value for average has now been set to `macro` instead.\")\n        if avg == \"binary\":\n            if self.pos_label not in (0, 1):\n                self.pos_label = 1\n                warn(\"Invalid value for pos_label. It has now been set to 1.\")\n            if self.pos_label == 1: return Tensor([0,1])\n            else: return Tensor([1,0])\n        elif avg == \"micro\": return self.cm.sum(dim=0) / self.cm.sum()\n        elif avg == \"macro\": return torch.ones((self.n_classes,)) / self.n_classes\n        elif avg == \"weighted\": return self.cm.sum(dim=1) / self.cm.sum()\n        \n    def _recall(self):\n        rec = torch.diag(self.cm) / (self.cm.sum(dim=1) + self.eps)\n        if self.average is None: return rec\n        else:\n            if self.average == \"micro\": weights = self._weights(avg=\"weighted\")\n            else: weights = self._weights(avg=self.average)\n            return (rec * weights).sum()\n    \n    def on_epoch_end(self, last_metrics, **kwargs): \n        return add_metrics(last_metrics, self._recall())\n    \nMetric_grapheme = partial(Metric_idx,0)\nMetric_vowel = partial(Metric_idx,1)\nMetric_consonant = partial(Metric_idx,2)\n\nclass Metric_tot(Callback):\n    def __init__(self):\n        super().__init__()\n        self.grapheme = Metric_idx(0)\n        self.vowel = Metric_idx(1)\n        self.consonant = Metric_idx(2)\n        \n    def on_epoch_begin(self, **kwargs):\n        self.grapheme.on_epoch_begin(**kwargs)\n        self.vowel.on_epoch_begin(**kwargs)\n        self.consonant.on_epoch_begin(**kwargs)\n    \n    def on_batch_end(self, last_output:Tensor, last_target:Tensor, **kwargs):\n        self.grapheme.on_batch_end(last_output, last_target, **kwargs)\n        self.vowel.on_batch_end(last_output, last_target, **kwargs)\n        self.consonant.on_batch_end(last_output, last_target, **kwargs)\n        \n    def on_epoch_end(self, last_metrics, **kwargs): \n        return add_metrics(last_metrics, 0.5*self.grapheme._recall() +\n                0.25*self.vowel._recall() + 0.25*self.consonant._recall())","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":false,"trusted":true},"cell_type":"code","source":"#fix the issue in fast.ai of saving gradients along with weights\n#so only weights are written, and files are ~4 times smaller\n\nclass SaveModelCallback(TrackerCallback):\n    \"A `TrackerCallback` that saves the model when monitored quantity is best.\"\n    def __init__(self, learn:Learner, monitor:str='valid_loss', mode:str='auto',\n                 every:str='improvement', name:str='bestmodel'):\n        super().__init__(learn, monitor=monitor, mode=mode)\n        self.every,self.name = every,name\n        if self.every not in ['improvement', 'epoch']:\n            warn(f'SaveModel every {self.every} is invalid, falling back to \"improvement\".')\n            self.every = 'improvement'\n                 \n    def jump_to_epoch(self, epoch:int)->None:\n        try: \n            self.learn.load(f'{self.name}_{epoch-1}', purge=False)\n            print(f\"Loaded {self.name}_{epoch-1}\")\n        except: print(f'Model {self.name}_{epoch-1} not found.')\n\n    def on_epoch_end(self, epoch:int, **kwargs:Any)->None:\n        \"Compare the value monitored to its best score and maybe save the model.\"\n        if self.every==\"epoch\": \n            #self.learn.save(f'{self.name}_{epoch}')\n            torch.save(learn.model.state_dict(),f'{self.name}_{epoch}.pth')\n        else: #every=\"improvement\"\n            current = self.get_monitor_value()\n            if current is not None and self.operator(current, self.best):\n                #print(f'Better model found at epoch {epoch} \\\n                #  with {self.monitor} value: {current}.')\n                self.best = current\n                #self.learn.save(f'{self.name}')\n                torch.save(learn.model.state_dict(),f'{self.name}.pth')\n\n    def on_train_end(self, **kwargs):\n        \"Load the best model.\"\n        if self.every==\"improvement\" and os.path.isfile(f'{self.name}.pth'):\n            #self.learn.load(f'{self.name}', purge=False)\n            self.model.load_state_dict(torch.load(f'{self.name}.pth'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# MixUp"},{"metadata":{},"cell_type":"markdown","source":"The code below modifies fast.ai MixUp calback to make it compatible with the current data."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"class MixUpLoss(Module):\n    \"Adapt the loss function `crit` to go with mixup.\"\n    \n    def __init__(self, crit, reduction='mean'):\n        super().__init__()\n        if hasattr(crit, 'reduction'): \n            self.crit = crit\n            self.old_red = crit.reduction\n            setattr(self.crit, 'reduction', 'none')\n        else: \n            self.crit = partial(crit, reduction='none')\n            self.old_crit = crit\n        self.reduction = reduction\n        \n    def forward(self, output, target):\n        if len(target.shape) == 2 and target.shape[1] == 7:\n            loss1, loss2 = self.crit(output,target[:,0:3].long()), self.crit(output,target[:,3:6].long())\n            d = loss1 * target[:,-1] + loss2 * (1-target[:,-1])\n        else:  d = self.crit(output, target)\n        if self.reduction == 'mean':    return d.mean()\n        elif self.reduction == 'sum':   return d.sum()\n        return d\n    \n    def get_old(self):\n        if hasattr(self, 'old_crit'):  return self.old_crit\n        elif hasattr(self, 'old_red'): \n            setattr(self.crit, 'reduction', self.old_red)\n            return self.crit\n\nclass MixUpCallback(LearnerCallback):\n    \"Callback that creates the mixed-up input and target.\"\n    def __init__(self, learn:Learner, alpha:float=0.4, stack_x:bool=False, stack_y:bool=True):\n        super().__init__(learn)\n        self.alpha,self.stack_x,self.stack_y = alpha,stack_x,stack_y\n    \n    def on_train_begin(self, **kwargs):\n        if self.stack_y: self.learn.loss_func = MixUpLoss(self.learn.loss_func)\n        \n    def on_batch_begin(self, last_input, last_target, train, **kwargs):\n        \"Applies mixup to `last_input` and `last_target` if `train`.\"\n        if not train: return\n        lambd = np.random.beta(self.alpha, self.alpha, last_target.size(0))\n        lambd = np.concatenate([lambd[:,None], 1-lambd[:,None]], 1).max(1)\n        lambd = last_input.new(lambd)\n        shuffle = torch.randperm(last_target.size(0)).to(last_input.device)\n        x1, y1 = last_input[shuffle], last_target[shuffle]\n        if self.stack_x:\n            new_input = [last_input, last_input[shuffle], lambd]\n        else: \n            out_shape = [lambd.size(0)] + [1 for _ in range(len(x1.shape) - 1)]\n            new_input = (last_input * lambd.view(out_shape) + x1 * (1-lambd).view(out_shape))\n        if self.stack_y:\n            new_target = torch.cat([last_target.float(), y1.float(), lambd[:,None].float()], 1)\n        else:\n            if len(last_target.shape) == 2:\n                lambd = lambd.unsqueeze(1).float()\n            new_target = last_target.float() * lambd + y1.float() * (1-lambd)\n        return {'last_input': new_input, 'last_target': new_target}  \n    \n    def on_train_end(self, **kwargs):\n        if self.stack_y: self.learn.loss_func = self.learn.loss_func.get_old()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Training"},{"metadata":{},"cell_type":"markdown","source":"I have performed a check of different optimizers and schedules on a [similar task](https://www.kaggle.com/c/Kannada-MNIST/discussion/122430), and [Over9000 optimizer](https://github.com/mgrankin/over9000) cosine annealing **without warm-up** worked the best. Freezing the backbone at the initial stage of training didn't give me any advantage in that test, so here I perform the training straight a way with discriminative learning rate (smaller lr for backbone)."},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"model = Dnet_1ch()\nlearn = Learner(data, model, loss_func=Loss_combine(), opt_func=Over9000,\n        metrics=[Metric_grapheme(),Metric_vowel(),Metric_consonant(),Metric_tot()])\nlogger = CSVLogger(learn,f'log{fold}')\nlearn.clip_grad = 1.0\nlearn.split([model.head1])\nlearn.unfreeze()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"model","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"learn.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.fit_one_cycle(32, max_lr=slice(0.2e-2,1e-2), wd=[1e-3,0.1e-1], pct_start=0.0, \n    div_factor=100, callbacks = [logger, SaveModelCallback(learn,monitor='metric_tot',\n    mode='max',name=f'model_{fold}'),MixUpCallback(learn)])\n#metrics: Metric_grapheme, Metric_vowel, Metric_consonant, Metric_tot (competition metric)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}