{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Motivation"},{"metadata":{},"cell_type":"markdown","source":"Although I'm a beginner in computer vision, I'd like to share a starter code by pytorch. I tried to follow the manner introduced in the pytorch [tutorials](https://pytorch.org/tutorials/) since I always suffered from understanding fully customized or wrapped codes shared in the kernels. I also customized a little since the example condes in these tutorials are too simple that they don't tell us how to implement early stopping, learning rate update with validation etc... Anyway, if there is better way of coding, please leave me a comment.\n\nI've refered the following datasets and notebooks:\nDataset\n- [Iterative-Stratification](https://www.kaggle.com/sheriytm/iterativestratification)\n- [EfficientNet PyTorch](https://www.kaggle.com/hmendonca/efficientnet-pytorch)\n- [EfficietNet3-Pytorch-Training-Inference](https://www.kaggle.com/gopidurgaprasad/efficietnet3-pytorch-training-inference)\n\nNotebooks\n- [iterative stratification](https://www.kaggle.com/yiheng/iterative-stratification)\n- [mixup/cutmix is all you need](https://www.kaggle.com/c/bengaliai-cv19/discussion/126504)"},{"metadata":{},"cell_type":"markdown","source":"# Install Necessary Libraries\n\nSince we cannot use the Internet access in this competition, the installer files of the necessary libraries need to be added as datasets"},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install ../input/efficientnet-pytorch/EfficientNet-PyTorch/EfficientNet-PyTorch-master/  > /dev/null\n#!pip install ../input/pretrainedmodels/pretrainedmodels-0.7.4/pretrainedmodels-0.7.4/ > /dev/null\n!pip install iterative-stratification > /dev/null","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Import Libraries"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"collapsed":true},"cell_type":"code","source":"#+---- Basic Libraries ----+#\nimport sys, os, time, gc, random\nfrom pathlib import Path\nimport pandas as pd\nimport numpy as np\nimport copy\n\n#+---- Utilities Libraries ----+#\nfrom efficientnet_pytorch import EfficientNet\nfrom iterstrat.ml_stratifiers import MultilabelStratifiedKFold\nfrom PIL import Image\nfrom tqdm.notebook import tqdm\nimport sklearn\n\n#+---- Pytorch Libraries ----+#\nimport torch\nimport torch.nn as nn\nimport torch.optim as optim\nfrom torch.optim import lr_scheduler\nfrom torch.utils import model_zoo\nimport numpy as np\nimport torchvision\nfrom torchvision import datasets, models, transforms\nfrom torch.utils.data import Dataset, DataLoader\nfrom torch.nn import functional as F\n\n#+---- List the input data ----+#\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"markdown","source":"# Initial Settings"},{"metadata":{},"cell_type":"markdown","source":"## Directories"},{"metadata":{"trusted":true},"cell_type":"code","source":"DATADIR = Path('/kaggle/input/bengaliai-cv19')\nFEATHERDIR = Path('/kaggle/input/bengaliaicv19feather')\nOUTDIR = Path('.')\nMDL_DIR = '/models'\nLOG_DIR = '/logs'\nif not os.path.exists(f'.{MDL_DIR}'):\n    os.mkdir(f'.{MDL_DIR}')\nif not os.path.exists(f'.{LOG_DIR}'):\n    os.mkdir(f'.{LOG_DIR}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Learning Parameters"},{"metadata":{"trusted":true},"cell_type":"code","source":"DEBUG =True # if we train model in small part of dataset to save time to debug\nSUBMISSION = True\nWORKER = 4\nSEED = 6666\n\nBATCH_SIZE =16 ## batch size is changed from 64 to 16 because of the RAM limitation\nNUM_EPOCH = 1\nIMAGE_SIZE=128\nDEVICE = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n\nMODEL_NAME ='efficientnet-b4'\nVER = 'fold_1_mixup_cutmix'\nN_Fold = 10 # 10-fold cross validation\nTRAIN_RATIO = 0.9 # if holdout for validation, the data size for validation dataset\nCV = True # choose validation methed: if you train model by cross validation, then True, otherwise False for hold out\nFold = 1 # clarify the fold to be trained\nPATIAENCE = 4 # early stopping patiance parameter","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"n_grapheme = 168\nn_vowel = 11\nn_consonant = 7\nn_total = n_grapheme + n_vowel + n_consonant","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data Preparation"},{"metadata":{},"cell_type":"markdown","source":"## Transform class for data preprocessing and augmentations"},{"metadata":{"trusted":true},"cell_type":"code","source":"def prepare_image(datadir, featherdir, data_type='train',\n                  submission=False, indices=[0, 1, 2, 3]):\n    assert data_type in ['train', 'test']\n    if submission:\n        image_df_list = [pd.read_parquet(datadir / f'{data_type}_image_data_{i}.parquet')\n                         for i in indices]\n    else:\n        image_df_list = [pd.read_feather(featherdir / f'{data_type}_image_data_{i}.feather')\n                         for i in indices]\n\n    print('image_df_list', len(image_df_list))\n    HEIGHT = 137\n    WIDTH = 236\n    images = [df.iloc[:, 1:].values.reshape(-1, HEIGHT, WIDTH) for df in image_df_list]\n    del image_df_list\n    gc.collect()\n    images = np.concatenate(images, axis=0)\n    return images","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def crop_char_image(image, threshold=5./255.):\n    assert image.ndim == 2\n    is_black = image > threshold\n\n    is_black_vertical = np.sum(is_black, axis=0) > 0\n    is_black_horizontal = np.sum(is_black, axis=1) > 0\n    left = np.argmax(is_black_horizontal)\n    right = np.argmax(is_black_horizontal[::-1])\n    top = np.argmax(is_black_vertical)\n    bottom = np.argmax(is_black_vertical[::-1])\n    height, width = image.shape\n    cropped_image = image[left:height - right, top:width - bottom]\n    return cropped_image","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data_transforms = {\n    'train': transforms.Compose([\n        transforms.ColorJitter(0.5,0.5,0.5,0.5),\n        transforms.RandomAffine(degrees=0.6),\n        transforms.Resize((IMAGE_SIZE,IMAGE_SIZE)),\n        transforms.ToTensor(),\n        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),\n    \n    ]),\n    'val': transforms.Compose([\n        transforms.Resize((IMAGE_SIZE,IMAGE_SIZE)),\n        transforms.ToTensor(),\n        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])\n    ]),\n    'test': transforms.Compose([\n        transforms.Resize((IMAGE_SIZE,IMAGE_SIZE)),\n        transforms.ToTensor(),\n        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])\n    ])\n}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Dataset Class"},{"metadata":{"trusted":true},"cell_type":"code","source":"# we need to prepare the dataset class by customizing the __len__ and __getitem__ funcitions\n# suit for the prediction task \nclass BengaliAIDataset(Dataset):\n    def __init__(self, images, labels=None, transform=None, indices=None):\n        self.transform = transform\n        self.images = images\n        self.labels = labels\n        if indices is None:\n            indices = np.arange(len(images))\n        self.indices = indices\n        self.train = labels is not None\n\n    def __len__(self):\n        \"\"\"return length of this dataset\"\"\"\n        return len(self.indices)\n      \n    def __getitem__(self, i):\n        \"\"\"Return i-th data\"\"\"\n        i = self.indices[i]\n        x = self.images[i]\n        # Opposite white and black: background will be white and\n        # for future Affine transformation\n        x = (255 - x).astype(np.float32) #/ 255.\n        x = crop_char_image(x)\n        x = Image.fromarray(x).convert(\"RGB\")\n        x = self.transform(x)\n        if self.train:\n            y = self.labels[i]\n            return x, y\n        else:\n            return x","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data Separation"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n# Split data set used for cross validation\ntrain = pd.read_csv(DATADIR/'train.csv')\ntrain['id'] = train['image_id'].apply(lambda x: int(x.split('_')[1]))\nX, y = train[['id', 'grapheme_root', 'vowel_diacritic', 'consonant_diacritic']]\\\n.values[:,0], train.values[:,1:]\ntrain['fold'] = np.nan\nmskf = MultilabelStratifiedKFold(n_splits=N_Fold)\nfor i, (_, index) in enumerate(mskf.split(X, y)):\n    #print('Fold '+str(i+1))\n    train.iloc[index, -1] = i\ntrain['fold'] = train['fold'].astype('int')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ntrain_labels = train[['grapheme_root', 'vowel_diacritic', 'consonant_diacritic']].values\nindices = [0] if DEBUG else [0, 1, 2, 3]\ntrain_images = prepare_image(\n    DATADIR, FEATHERDIR, data_type='train', submission=False, indices=indices)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"n_dataset = len(train_images)\n\nif not CV:\n    train_data_size = 200 if DEBUG else int(n_dataset * TRAIN_RATIO)\n    valid_data_size = 100 if DEBUG else int(n_dataset - train_data_size)\n    perm = np.random.RandomState(777).permutation(n_dataset)\n    print('perm', perm)\n\n    train_dataset = BengaliAIDataset(\n        train_images, train_labels, transform=data_transforms['train'],\n        indices=perm[:train_data_size])\n\n    valid_dataset = BengaliAIDataset(\n        train_images, train_labels, transform=data_transforms['val'],\n        indices=perm[train_data_size:train_data_size+valid_data_size])\nelse:\n    valid_idx = np.array(train[train['fold']==Fold].index)\n    trn_idx = np.array(train[train['fold']!=Fold].index)\n    trn_idx = trn_idx[:200] if DEBUG else trn_idx\n    valid_idx = valid_idx[:100] if DEBUG else valid_idx\n    \n    train_dataset = BengaliAIDataset(\n        train_images, train_labels, transform=data_transforms['train'],\n        indices=trn_idx)\n    valid_dataset = BengaliAIDataset(\n        train_images, train_labels, transform=data_transforms['val'],\n        indices=valid_idx)\n\ntrain_loader = DataLoader(train_dataset, batch_size=BATCH_SIZE, shuffle=True, num_workers=WORKER)\nvalid_loader = DataLoader(valid_dataset, batch_size=BATCH_SIZE, shuffle=False, num_workers=WORKER)\n\ndataloaders = {'train':train_loader, 'val': valid_loader}\ndataset_sizes = {'train':len(train_dataset), 'val': len(valid_dataset)}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"image, label = train_dataset[1]\nprint('image', image.shape, 'label', label)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Functions for evaluation and augmentations"},{"metadata":{"trusted":true},"cell_type":"code","source":"def macro_recall(pred_labels, y, n_grapheme=168, n_vowel=11, n_consonant=7):\n    recall_grapheme = sklearn.metrics.recall_score(y[0], pred_labels[0],  average='macro')\n    recall_vowel = sklearn.metrics.recall_score(y[1],pred_labels[1], average='macro')\n    recall_consonant = sklearn.metrics.recall_score(y[2],pred_labels[2],  average='macro')\n    scores = [recall_grapheme, recall_vowel, recall_consonant]\n    final_score = np.average(scores, weights=[2, 1, 1])\n    print(f'recall: grapheme {recall_grapheme}, vowel {recall_vowel}, consonant {recall_consonant}, '\n           f'total {final_score}')\n    return final_score","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_pred(preds_list, label_list):\n    #preds_list is torch tensor to device\n    #label_list is torch tensor to device\n    _, pred0 = torch.max(preds_list[0], 1)\n    _, pred1 = torch.max(preds_list[1], 1)\n    _, pred2 = torch.max(preds_list[2], 1)\n    p0 = pred0.cpu().numpy()\n    p1 = pred1.cpu().numpy()\n    p2 = pred2.cpu().numpy()\n    pred_labels = [p0, p1, p2]\n    #print(pred_labels)\n    a0 = label_list[0].cpu().numpy()\n    a1 = label_list[1].cpu().numpy()\n    a2 = label_list[2].cpu().numpy() \n    y = [a0, a1, a2]\n    #print(y)\n    return pred_labels, y","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def rand_bbox(size, lam):\n    W = size[2]\n    H = size[3]\n    cut_rat = np.sqrt(1. - lam)\n    cut_w = np.int(W * cut_rat)\n    cut_h = np.int(H * cut_rat)\n\n    # uniform\n    cx = np.random.randint(W)\n    cy = np.random.randint(H)\n\n    bbx1 = np.clip(cx - cut_w // 2, 0, W)\n    bby1 = np.clip(cy - cut_h // 2, 0, H)\n    bbx2 = np.clip(cx + cut_w // 2, 0, W)\n    bby2 = np.clip(cy + cut_h // 2, 0, H)\n\n    return bbx1, bby1, bbx2, bby2\ndef cutmix(data, targets1, targets2, targets3, alpha):\n    indices = torch.randperm(data.size(0))\n    shuffled_data = data[indices]\n    shuffled_targets1 = targets1[indices]\n    shuffled_targets2 = targets2[indices]\n    shuffled_targets3 = targets3[indices]\n\n    lam = np.random.beta(alpha, alpha)\n    bbx1, bby1, bbx2, bby2 = rand_bbox(data.size(), lam)\n    data[:, :, bbx1:bbx2, bby1:bby2] = data[indices, :, bbx1:bbx2, bby1:bby2]\n    # adjust lambda to exactly match pixel ratio\n    lam = 1 - ((bbx2 - bbx1) * (bby2 - bby1) / (data.size()[-1] * data.size()[-2]))\n\n    targets = [targets1, shuffled_targets1, targets2, shuffled_targets2, targets3, \\\n               shuffled_targets3, lam]\n    return data, targets\n\ndef mixup(data, targets1, targets2, targets3, alpha):\n    indices = torch.randperm(data.size(0))\n    shuffled_data = data[indices]\n    shuffled_targets1 = targets1[indices]\n    shuffled_targets2 = targets2[indices]\n    shuffled_targets3 = targets3[indices]\n\n    lam = np.random.beta(alpha, alpha)\n    data = data * lam + shuffled_data * (1 - lam)\n    targets = [targets1, shuffled_targets1, targets2, shuffled_targets2, targets3,\\\n               shuffled_targets3, lam]\n\n    return data, targets\n\n\ndef cutmix_criterion(preds1,preds2,preds3, targets):\n    targets1, targets2,targets3, targets4,targets5, targets6, lam = targets[0], targets[1],\\\n    targets[2], targets[3], targets[4], targets[5], targets[6]\n    criterion = nn.CrossEntropyLoss(reduction='mean')\n    return lam * criterion(preds1, targets1) + (1 - lam) * criterion(preds1, targets2) +\\\nlam * criterion(preds2, targets3) + (1 - lam) * criterion(preds2, targets4) +\\\nlam * criterion(preds3, targets5) + (1 - lam) * criterion(preds3, targets6)\n\ndef mixup_criterion(preds1,preds2,preds3, targets):\n    targets1, targets2,targets3, targets4,targets5, targets6, lam = targets[0], \\\n    targets[1], targets[2], targets[3], targets[4], targets[5], targets[6]\n    criterion = nn.CrossEntropyLoss(reduction='mean')\n    return lam * criterion(preds1, targets1) + (1 - lam) * criterion(preds1, targets2)\\\n+ lam * criterion(preds2, targets3) + (1 - lam) * criterion(preds2, targets4) +\\\nlam * criterion(preds3, targets5) + (1 - lam) * criterion(preds3, targets6)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Functions for model training"},{"metadata":{},"cell_type":"markdown","source":"There are train phase and validation phase in each epoch. During the train phase, the 5% and 5% of the mini-batch of the data are augumented by mixup or cutmix. The hyper-parameter alpha is set to 0.1 since too large alpha may cause [underfitting](https://arxiv.org/abs/1710.09412).\n```\nratio = 0.1\nrandomlist = random.sample( range(length), int(ratio*length))\nmixuplist = randomlist[:int(ratio*length/2)]\ncutmixlist = randomlist[int(ratio*length/2):]\n```\nIn the following train_model function, the model is trained based on the loss function, but early stopping and learning rate scheduler work based the recall score of the validation dataset."},{"metadata":{"trusted":true},"cell_type":"code","source":"def train_model(model, dataloaders, criterion, optimizer, scheduler,start_epoch,\n                num_epochs, device, patiance):\n    since = time.time()\n    \n    trn_loss_list =[]\n    trn_acc_list = []\n    val_loss_list =[]\n    val_acc_list = []\n    epoch_list = []\n    recall_list = []\n\n    best_model_wts = copy.deepcopy(model.state_dict())\n    best_acc = 0.0\n    best_loss = 10\n    best_recall = 0.0\n    torch.backends.cudnn.benchmark = True\n    early_stopping_counter = 0\n    \n    for epoch in range(num_epochs)[start_epoch:]:\n        print('Epoch {}/{}'.format(epoch+1, num_epochs))\n        print('-' * 30)  \n        \n        if early_stopping_counter == patiance:\n            print(f'Early Stopped since loss have not decreased for {patiance} epoch.')\n            break\n        epoch_list.append(epoch+1)\n        # Each epoch has a training and validation phase\n        for phase in ['train', 'val']:\n            if phase == 'train':\n                model.train()  # Set model to training mode\n            else:\n                model.eval()   # Set model to evaluate mode\n\n            epoch_loss = 0.0\n            epoch_corrects = 0\n            dataset_sizes = len(dataloaders[phase].dataset)\n            length = int(np.floor(dataset_sizes/BATCH_SIZE))\n            ratio = 0.1\n            randomlist = random.sample( range(length), int(ratio*length))\n            mixuplist = randomlist[:int(ratio*length/2)]\n            cutmixlist = randomlist[int(ratio*length/2):]\n\n            # Iterate over data.\n            for idx, (inputs, labels) in enumerate(tqdm(dataloaders[phase])):\n                #print(inputs.shape)\n                inputs = inputs.to(device)\n                labels = labels.transpose(1,0).to(device) #use when single label for one image\n\n                grapheme_root = labels[0]\n                vowel_diacritic = labels[1]\n                consonant_diacritic = labels[2]\n                \n                # zero the parameter gradients\n                optimizer.zero_grad()\n\n                # Forward\n                # track history if only in train\n                with torch.set_grad_enabled(phase == 'train'):                                 \n                    if phase == 'train':\n                        if idx in mixuplist:\n                            \n                            inputs_mixed, labels_mixed = mixup(inputs, grapheme_root, \\\n                                                               vowel_diacritic, consonant_diacritic, 0.1)\n\n                            outputs = model(inputs_mixed) \n                            grapheme_root_prd = outputs[0]\n                            vowel_diacritic_prd = outputs[1]\n                            consonant_diacritic_prd = outputs[2]\n                            loss = mixup_criterion(grapheme_root_prd,vowel_diacritic_prd,\\\n                                               consonant_diacritic_prd, labels_mixed)\n                            \n                        elif idx in cutmixlist:\n                            \n                            inputs_cut, labels_cut = cutmix(inputs, grapheme_root, \\\n                                                               vowel_diacritic, consonant_diacritic, 0.1)\n\n                            outputs = model(inputs_cut) \n                            grapheme_root_prd = outputs[0]\n                            vowel_diacritic_prd = outputs[1]\n                            consonant_diacritic_prd = outputs[2]\n                            loss = cutmix_criterion(grapheme_root_prd,vowel_diacritic_prd,\\\n                                               consonant_diacritic_prd, labels_cut)\n                        \n                        else:\n                            outputs = model(inputs)\n                            grapheme_root_prd = outputs[0]\n                            vowel_diacritic_prd = outputs[1]\n                            consonant_diacritic_prd = outputs[2]\n                            loss = (1/3)*(criterion(grapheme_root_prd, grapheme_root)+\\\n                                  criterion(vowel_diacritic_prd, vowel_diacritic) +\\\n                                     criterion(consonant_diacritic_prd, consonant_diacritic))\n                        loss.backward()\n                        optimizer.step()\n                    if phase == 'val':\n                        outputs = model(inputs)\n                        grapheme_root_prd = outputs[0]\n                        vowel_diacritic_prd = outputs[1]\n                        consonant_diacritic_prd = outputs[2]\n                        loss = (1/3)*(criterion(grapheme_root_prd, grapheme_root)+\\\n                              criterion(vowel_diacritic_prd, vowel_diacritic) +\\\n                                 criterion(consonant_diacritic_prd, consonant_diacritic))\n\n                        \n                # statistics: inputs.size(0) is batch size\n                epoch_loss += loss.item() * inputs.size(0) # total loss for this batch\n                epoch_corrects += torch.sum(torch.max(outputs[0], 1)[1] == labels[0])+\\\n                    torch.sum(torch.max(outputs[1], 1)[1] == labels[1])+\\\n                    torch.sum(torch.max(outputs[2], 1)[1] == labels[2])\n                \n            epoch_loss = epoch_loss / dataset_sizes\n            epoch_acc = epoch_corrects.double() / (dataset_sizes*3)\n            pred, lbls = get_pred(outputs, labels)\n            recall = macro_recall(pred, lbls, \\\n                                      n_grapheme=168, n_vowel=11, n_consonant=7)\n            \n            if phase == 'train':\n                trn_loss_list.append(epoch_loss)\n                trn_acc_list.append(epoch_acc.cpu().numpy())\n\n            print('{} Loss: {:.4f} Acc: {:.4f}'.format(\n                phase, epoch_loss, epoch_acc))\n\n            # Deep copy the model\n            if phase == 'val' and recall > best_recall:\n                best_model_wts = copy.deepcopy(model.state_dict())\n                if not os.path.exists(f'.{MDL_DIR}/{MODEL_NAME}_{VER}'):\n                    os.mkdir(f'.{MDL_DIR}/{MODEL_NAME}_{VER}')\n                save_path = f'.{MDL_DIR}/{MODEL_NAME}_{VER}/{MODEL_NAME}_'+str(epoch+1)+'.pth'\n                torch.save(model_ft.state_dict(),save_path)\n                best_epoch = epoch\n            \n            if phase == 'val':\n                if epoch == 0 or epoch == start_epoch:\n                    best_recall = recall\n                    \n                else:\n                    if recall > best_recall:\n                        print(recall)\n                        best_recall = recall\n                        early_stopping_counter = 0\n                    else:\n                        early_stopping_counter += 1\n                        print(f'Early stopping counter: {early_stopping_counter}')\n\n                scheduler.step(epoch_loss) \n                val_loss_list.append(epoch_loss)\n                val_acc_list.append(epoch_acc.cpu().numpy())\n              \n                \n                print('valid recall score is {:.3f}'.format(recall))\n                recall_list.append(recall)\n\n        \n    time_elapsed = time.time() - since\n    print('Training complete in {:.0f}m {:.0f}s'.format(\n        time_elapsed // 60, time_elapsed % 60))\n    print('Best val Recall: {:4f}'.format(best_recall))\n\n    # load best model weights\n    model.load_state_dict(best_model_wts)\n    if not os.path.exists(f'.{LOG_DIR}/log_{MODEL_NAME}_{VER}.csv'):\n        log = pd.DataFrame()\n        log['Epoch'] = epoch_list\n        log['Train Loss'] = trn_loss_list\n        log['Train Acc'] = trn_acc_list\n        log['Valid Loss'] = val_loss_list\n        log['Valid Acc'] = val_acc_list\n        log['Recall'] = recall_list\n        log.to_csv(f'.{LOG_DIR}/log_{MODEL_NAME}_{VER}.csv',index=False)\n    else:\n        log = pd.DataFrame()\n        log['Epoch'] = epoch_list\n        log['Train Loss'] = trn_loss_list\n        log['Train Acc'] = trn_acc_list\n        log['Valid Loss'] = val_loss_list\n        log['Valid Acc'] = val_acc_list\n        log['Recall'] = recall_list\n        log_old = pd.read_csv(f'.{LOG_DIR}/log_{MODEL_NAME}_{VER}.csv')\n        LOG = pd.concat([log_old, log], axis=0)\n        LOG.reset_index(drop=True, inplace=True)\n        LOG.to_csv(f'.{LOG_DIR}/log_{MODEL_NAME}_{VER}.csv',index=False)\n    return model, best_epoch+1","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Since the last layer of the original efficient net is for singla class prediction, we need to customize a litte. The modifined model will give the 3 ouptuts for grapheme, consonant, and vowel."},{"metadata":{"trusted":true},"cell_type":"code","source":"class bengali_model(nn.Module):\n    def __init__(self, num_classes1, num_classes2, num_classes3):\n        super(bengali_model, self).__init__()\n        #pretrain models\n        #self.model = pretrainedmodels.__dict__[MODEL_NAME](pretrained=None)\n        #num_ftrs = self.model.last_linear.in_features\n        #self.model.last_linear = nn.Identity()\n        \n        # EfficientNet\n        self.model = EfficientNet.from_name(MODEL_NAME)\n        \n        # if internet is allowed, we can use pretrained weight\n        #self.model = EfficientNet.from_pretrained(MODEL_NAME)\n        \n        num_ftrs = 1792\n        \n        self.fc1 = nn.Linear(num_ftrs, num_classes1)\n        self.fc2 = nn.Linear(num_ftrs, num_classes2)\n        self.fc3 = nn.Linear(num_ftrs, num_classes3)\n\n    def forward(self, x):\n        #x = self.model(x) #pretrain models\n        bs, _, _, _ = x.shape\n        x = self.model.extract_features(x)\n        x = F.adaptive_avg_pool2d(x, 1).reshape(bs, -1)\n        out1 = self.fc1(x)\n        out2 = self.fc2(x)\n        out3 = self.fc3(x)\n        return out1, out2, out3","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# --- Model --- Stage 1\nmodel_ft = bengali_model(n_grapheme, n_vowel, n_consonant)\nmodel_ft = model_ft.to(DEVICE)\ncriterion = nn.CrossEntropyLoss()\noptimizer = torch.optim.Adam(model_ft.parameters(), lr=0.001)\nscheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(\n    optimizer, mode='min', factor=0.5, patience=2, min_lr=1e-10, verbose=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true},"cell_type":"code","source":"torch.cuda.empty_cache()\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"START_EPOCH = 0\nmodel_ft, best_epoch = train_model(model_ft, dataloaders, criterion, optimizer, scheduler, START_EPOCH,\n                                   NUM_EPOCH, DEVICE, PATIAENCE)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Evaluation"},{"metadata":{"trusted":true},"cell_type":"code","source":"def predict(model, dataloaders, phase, device):\n    model.eval()\n    output_list = []\n    label_list = []\n    with torch.no_grad():\n        if phase == 'test':\n            for i, inputs in enumerate(tqdm(dataloaders)):\n                \n                inputs = inputs.to(device)\n                outputs = model(inputs)\n                _, pred0 = torch.max(outputs[0], 1)\n                _, pred1 = torch.max(outputs[1], 1)\n                _, pred2 = torch.max(outputs[2], 1)\n                preds = (pred0, pred1, pred2)\n                output_list.append(preds)\n            return output_list\n        elif phase == 'val':\n            for i, (inputs, labels) in enumerate(tqdm(dataloaders)):\n                \n                inputs = inputs.to(device)\n                outputs = model(inputs)\n                _, pred0 = torch.max(outputs[0], 1)\n                _, pred1 = torch.max(outputs[1], 1)\n                _, pred2 = torch.max(outputs[2], 1)\n                preds = (pred0, pred1, pred2)\n                output_list.append(preds)\n                label_list.append(labels.transpose(1,0))\n            return output_list, label_list","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"save_path = f'.{MDL_DIR}/{MODEL_NAME}_{VER}/{MODEL_NAME}_'+str(best_epoch)+'.pth'\nload_weights = torch.load(save_path)\nmodel_ft.load_state_dict(load_weights)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# --- Prediction ---\ndata_type = 'val'\nvalid_preds_list = []\nprint('valid_dataset', len(valid_dataset))\nvalid_preds_list, valid_label_list = predict(model_ft, valid_loader, data_type, DEVICE)\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Each test_preds indicates the prediction outputs of different batch\np0 = np.concatenate([valid_preds[0].cpu().numpy() for valid_preds in valid_preds_list], axis=0)\np1 = np.concatenate([valid_preds[1].cpu().numpy() for valid_preds in valid_preds_list], axis=0)\np2 = np.concatenate([valid_preds[2].cpu().numpy() for valid_preds in valid_preds_list], axis=0)\nprint('p0', p0.shape, 'p1', p1.shape, 'p2', p2.shape)\n\na0 = np.concatenate([valid_label[0].cpu().numpy() for valid_label in valid_label_list], axis=0)\na1 = np.concatenate([valid_label[1].cpu().numpy() for valid_label in valid_label_list], axis=0)\na2 = np.concatenate([valid_label[2].cpu().numpy() for valid_label in valid_label_list], axis=0)\nprint('a0', a0.shape, 'a1', a1.shape, 'a2', a2.shape)\n\npred_labels = [p0, p1, p2]\ny = [a0, a1, a2]\nmacro_recall(pred_labels, y, n_grapheme=168, n_vowel=11, n_consonant=7)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Inference"},{"metadata":{"trusted":true},"cell_type":"code","source":"def prepare_image_test(datadir, featherdir, data_type='train',\n                  submission=True, indices=[0, 1, 2, 3]):\n    assert data_type in ['train', 'test']\n    if submission:\n        image_df_list = [pd.read_parquet(datadir / f'{data_type}_image_data_{i}.parquet')\n                         for i in indices]\n        \n        \n    else:\n        image_df_list = [pd.read_feather(featherdir / f'{data_type}_image_data_{i}.feather')\n                         for i in indices]\n\n    print('image_df_list', len(image_df_list))\n    HEIGHT = 137\n    WIDTH = 236\n    images = [df.iloc[:, 1:].values.reshape(-1, HEIGHT, WIDTH) for df in image_df_list]\n    #del image_df_list\n    gc.collect()\n    images = np.concatenate(images, axis=0)\n    return images, image_df_list","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# --- Prediction ---\n\nsts = time.time()\ndata_type = 'test'\ncomponents = ['consonant_diacritic', 'grapheme_root', 'vowel_diacritic']\ntarget=[] # model predictions placeholder\nrow_id=[] # row_id place holder\n\nfor i in tqdm(range(4)):\n    indices = [i]\n    test_images, df_test_img = prepare_image_test(\n        DATADIR, FEATHERDIR, data_type = data_type, submission=SUBMISSION, indices=indices)\n    n_dataset = len(test_images)\n    print(f'i={i}, n_dataset={n_dataset}')\n    test_dataset = BengaliAIDataset(\n    test_images, None,\n    transform=data_transforms[data_type])\n    print('test_dataset', len(test_dataset))\n    test_loader = DataLoader(test_dataset, batch_size=BATCH_SIZE, shuffle=False, num_workers=WORKER)\n    \n    test_preds = predict(model_ft, test_loader, data_type,DEVICE)\n    p0 = np.concatenate([valid_preds[0].cpu().numpy() for valid_preds in valid_preds_list], axis=0) #grapheme\n    p1 = np.concatenate([valid_preds[1].cpu().numpy() for valid_preds in valid_preds_list], axis=0) #vowel\n    p2 = np.concatenate([valid_preds[2].cpu().numpy() for valid_preds in valid_preds_list], axis=0) #consonant\n    tgt = [p2, p0, p1]\n    for idx, id in enumerate(df_test_img[0].image_id.values): # df_test_img.index.values has the test image_ids\n        #print(idx)\n        for i,comp in enumerate(components):\n            id_sample=id+'_'+comp\n            row_id.append(id_sample)\n            target.append(tgt[i].flatten()[idx]) # our model is a random integer generator between 0 and n_cls\n    del test_images, df_test_img\n    gc.collect()\n    if DEBUG:\n        break\ned = time.time()\nprint('Predicted in {:.3f} min'.format((ed-sts)/60))\ndel p0, p1, p2","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# create a dataframe with the solutions \nsub_df = pd.DataFrame(\n    {'row_id': row_id,\n    'target':target\n    },\n    columns =['row_id','target'] \n)\nsub_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub_df.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub = pd.read_csv('/kaggle/input/bengaliai-cv19/sample_submission.csv')\nif len(sub) == len(sub_df):\n    sub_df.to_csv('submission.csv', index=False)\n    print('Prediction file saved successfully.')\nelse:\n    sub.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}