{"cells":[{"metadata":{},"cell_type":"markdown","source":"This kernel is based on [Heng's Starter code](https://www.kaggle.com/c/bengaliai-cv19/discussion/123757). I have already published a [kernel](https://www.kaggle.com/bibek777/heng-starter-inference-kernel) doing the inference using his models. In this kernel, we will use his codes to do the training.\n\nI have uploaded the necessary codes from Heng's starter and opensourced it as a [kaggle dataset](https://www.kaggle.com/bibek777/hengcodes) so that you can use it to train in kaggle kernels/google-colabs. "},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport sys\nimport os\nfrom sklearn.model_selection import train_test_split\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"os.listdir('../input')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Add heng's code to our envionment and import the modules"},{"metadata":{"trusted":true},"cell_type":"code","source":"hengs_path = '../input/hengcodes'\nsys.path.append(hengs_path)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from common  import *\nfrom model   import *\nfrom kaggle import *","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Heng uses his own version of datasplit(which I have uploaded [here](https://www.kaggle.com/bibek777/hengdata)) in his codes. I tried using it but get memory error, maybe it's too large to load. So I have edited the dataloader in his code and use different split for train and valid dataset. The codes/ideas for dataloader is taken from this [kernel](https://www.kaggle.com/backaggle/catalyst-baseline). Also the dataset used in this kernel is taken from [here](https://www.kaggle.com/pestipeti/bengaliai), uploaded by Peter"},{"metadata":{"trusted":true},"cell_type":"code","source":"data_root = \"../input/bengaliai/256_train/256/\"\ndf = pd.read_csv(\"../input/bengaliai-cv19/train.csv\")\ntrain_df, valid_df = train_test_split(df, test_size=0.2, shuffle=True, random_state=2411)\n\nTASK_NAME = [ 'grapheme_root', 'vowel_diacritic', 'consonant_diacritic' ]\nNUM_TASK = len(TASK_NAME)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class KaggleDataset(Dataset):\n\n    def __init__(self, df, data_path, augment=None):\n        self.image_ids = df['image_id'].values\n        self.grapheme_roots = df['grapheme_root'].values\n        self.vowel_diacritics = df['vowel_diacritic'].values\n        self.consonant_diacritics = df['consonant_diacritic'].values\n\n        self.data_path = data_path\n        self.augment = augment\n\n    def __str__(self):\n        string  = ''\n        string += '\\tlen = %d\\n'%len(self)\n        string += '\\n'\n        return string\n\n\n    def __len__(self):\n        return len(self.image_ids)\n\n\n    def __getitem__(self, index):\n        # print(index)\n        image_id = self.image_ids[index]\n        grapheme_root = self.grapheme_roots[index]\n        vowel_diacritic = self.vowel_diacritics[index]\n        consonant_diacritic = self.consonant_diacritics[index]\n\n        image_id = os.path.join(self.data_path, image_id + '.png')\n\n        image = cv2.imread(image_id, 0)\n        image = cv2.cvtColor(image,cv2.COLOR_GRAY2BGR)\n        image = image.astype(np.float32)/255\n        label = [grapheme_root, vowel_diacritic, consonant_diacritic]\n\n        infor = Struct(\n            index    = index,\n            image_id = image_id,\n        )\n\n        if self.augment is None:\n            return image, label, infor\n        else:\n            return self.augment(image, label, infor)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def null_collate(batch):\n    batch_size = len(batch)\n\n    input = []\n    label = []\n    infor = []\n    for b in range(batch_size):\n        input.append(batch[b][0])\n        label.append(batch[b][1])\n        infor.append(batch[b][-1])\n\n    input = np.stack(input)\n    #input = input[...,::-1].copy()\n    input = input.transpose(0,3,1,2)\n\n    label = np.stack(label)\n\n    #----\n    input = torch.from_numpy(input).float()\n    truth = torch.from_numpy(label).long()\n    truth0, truth1, truth2 = truth[:,0],truth[:,1],truth[:,2]\n    truth = [truth0, truth1, truth2]\n    return input, truth, infor\n\n\n##############################################################\n\ndef tensor_to_image(tensor):\n    image = tensor.data.cpu().numpy()\n    image = image.transpose(0,2,3,1)\n    #image = image[...,::-1]\n    return image\n\n\n##############################################################\n\ndef do_random_crop_rotate_rescale(\n    image,\n    mode={'rotate': 10,'scale': 0.1,'shift': 0.1}\n):\n\n    dangle = 0\n    dscale_x, dscale_y = 0,0\n    dshift_x, dshift_y = 0,0\n\n    for k,v in mode.items():\n        if   'rotate'== k:\n            dangle = np.random.uniform(-v, v)\n        elif 'scale' == k:\n            dscale_x, dscale_y = np.random.uniform(-1, 1, 2)*v\n        elif 'shift' == k:\n            dshift_x, dshift_y = np.random.uniform(-1, 1, 2)*v\n        else:\n            raise NotImplementedError\n\n    #----\n\n    height, width = image.shape[:2]\n\n    cos = np.cos(dangle/180*PI)\n    sin = np.sin(dangle/180*PI)\n    sx,sy = 1 + dscale_x, 1+ dscale_y #1,1 #\n    tx,ty = dshift_x*width, dshift_y*height\n\n    src = np.array([[-width/2,-height/2],[ width/2,-height/2],[ width/2, height/2],[-width/2, height/2]], np.float32)\n    src = src*[sx,sy]\n    x = (src*[cos,-sin]).sum(1)+width/2 +tx\n    y = (src*[sin, cos]).sum(1)+height/2+ty\n    src = np.column_stack([x,y])\n\n    dst = np.array([[0,0],[width,0],[width,height],[0,height]])\n    s = src.astype(np.float32)\n    d = dst.astype(np.float32)\n    transform = cv2.getPerspectiveTransform(s,d)\n    image = cv2.warpPerspective( image, transform, (width, height), flags=cv2.INTER_LINEAR,\n                                 borderMode=cv2.BORDER_CONSTANT, borderValue=(1,1,1))\n\n    return image\n\n\ndef do_random_log_contast(image, gain=[0.70, 1.30] ):\n    gain = np.random.uniform(gain[0],gain[1],1)\n    inverse = np.random.choice(2,1)\n\n    if inverse==0:\n        image = gain*np.log(image+1)\n    else:\n        image = gain*(2**image-1)\n\n    image = np.clip(image,0,1)\n    return image\n\n\n#https://github.com/albumentations-team/albumentations/blob/8b58a3dbd2f35558b3790a1dbff6b42b98e89ea5/albumentations/augmentations/transforms.py\ndef do_grid_distortion(image, distort=0.25, num_step = 10):\n\n    # http://pythology.blogspot.sg/2014/03/interpolation-on-regular-distorted-grid.html\n    distort_x = [1 + random.uniform(-distort,distort) for i in range(num_step + 1)]\n    distort_y = [1 + random.uniform(-distort,distort) for i in range(num_step + 1)]\n\n    #---\n    height, width = image.shape[:2]\n    xx = np.zeros(width, np.float32)\n    step_x = width // num_step\n\n    prev = 0\n    for i, x in enumerate(range(0, width, step_x)):\n        start = x\n        end   = x + step_x\n        if end > width:\n            end = width\n            cur = width\n        else:\n            cur = prev + step_x * distort_x[i]\n        xx[start:end] = np.linspace(prev, cur, end - start)\n        prev = cur\n\n\n    yy = np.zeros(height, np.float32)\n    step_y = height // num_step\n\n    prev = 0\n    for idx, y in enumerate(range(0, height, step_y)):\n        start = y\n        end = y + step_y\n        if end > height:\n            end = height\n            cur = height\n        else:\n            cur = prev + step_y * distort_y[idx]\n\n        yy[start:end] = np.linspace(prev, cur, end - start)\n        prev = cur\n\n    map_x, map_y = np.meshgrid(xx, yy)\n    map_x = map_x.astype(np.float32)\n    map_y = map_y.astype(np.float32)\n    image = cv2.remap(image, map_x, map_y, interpolation=cv2.INTER_LINEAR, borderMode=cv2.BORDER_CONSTANT, borderValue=(1,1,1))\n\n    return image\n\n\n\n# ##---\n# #https://github.com/chainer/chainercv/blob/master/chainercv/links/model/ssd/transforms.py\ndef do_random_contast(image, alpha=[0,1]):\n    beta  = 0\n    alpha = random.uniform(*alpha) + 1\n    image = image.astype(np.float32) * alpha + beta\n    image = np.clip(image,0,1)\n    return image\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"################################################################################################\n\ndef train_augment(image, label, infor):\n    if np.random.rand()<0.5:\n        image = do_random_crop_rotate_rescale(image, mode={'rotate': 17.5,'scale': 0.25,'shift': 0.08})\n    if np.random.rand()<0.5:\n        image = do_grid_distortion(image, distort=0.20, num_step = 10)\n    return image, label, infor\n\n\n\ndef valid_augment(image, label, infor):\n    return image, label, infor","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#------------------------------------\ndef do_valid(net, valid_loader, out_dir=None):\n\n    valid_loss = np.zeros(6, np.float32)\n    valid_num  = np.zeros_like(valid_loss)\n\n    valid_probability = [[],[],[],]\n    valid_truth = [[],[],[],]\n\n    for t, (input, truth, infor) in enumerate(valid_loader):\n\n        #if b==5: break\n        batch_size = len(infor)\n\n        net.eval()\n        input = input.cuda()\n        truth = [t.cuda() for t in truth]\n\n        with torch.no_grad():\n            logit = data_parallel(net, input) #net(input)\n            probability = logit_to_probability(logit)\n\n            loss = criterion(logit, truth)\n            correct = metric(probability, truth)\n\n        #---\n        loss = [l.item() for l in loss]\n        l = np.array([ *loss, *correct, ])*batch_size\n        n = np.array([ 1, 1, 1, 1, 1, 1  ])*batch_size\n        valid_loss += l\n        valid_num  += n\n\n        #---\n        for i in range(NUM_TASK):\n            valid_probability[i].append(probability[i].data.cpu().numpy())\n            valid_truth[i].append(truth[i].data.cpu().numpy())\n\n        #print(valid_loss)\n        print('\\r %8d /%d'%(valid_num[0], len(valid_loader.dataset)),end='',flush=True)\n\n        pass  #-- end of one data loader --\n    assert(valid_num[0] == len(valid_loader.dataset))\n    valid_loss = valid_loss/(valid_num+1e-8)\n\n    #------\n    for i in range(NUM_TASK):\n        valid_probability[i] = np.concatenate(valid_probability[i])\n        valid_truth[i] = np.concatenate(valid_truth[i])\n    recall, avgerage_recall = compute_kaggle_metric(valid_probability, valid_truth)\n\n\n    return valid_loss, (recall, avgerage_recall)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def run_train():\n    out_dir = '/kaggle/working'\n    initial_checkpoint = None\n\n    schduler = NullScheduler(lr=0.01)\n    iter_accum = 1\n    batch_size = 64 #8\n\n    ## setup  -----------------------------------------------------------------------------\n    for f in ['checkpoint','train','valid'] : os.makedirs(out_dir +'/'+f, exist_ok=True)\n        \n    log = Logger()\n    log.open(out_dir+'/log.train.txt',mode='a')\n    log.write('\\n--- [START %s] %s\\n\\n' % (IDENTIFIER, '-' * 64))\n    log.write('\\t%s\\n' % COMMON_STRING)\n    log.write('\\n')\n\n    log.write('\\tSEED         = %u\\n' % SEED)\n    log.write('\\tout_dir      = %s\\n' % out_dir)\n    log.write('\\n')\n\n\n    ## dataset ----------------------------------------\n    log.write('** dataset setting **\\n')\n\n    train_dataset = KaggleDataset(\n        df = train_df, \n        data_path = data_root,\n        augment = train_augment,\n    )\n    train_loader  = DataLoader(\n        train_dataset,\n        sampler     = RandomSampler(train_dataset),\n        batch_size  = batch_size,\n        drop_last   = True,\n        num_workers = 0,\n        pin_memory  = True,\n        collate_fn  = null_collate\n    )\n\n\n    valid_dataset = KaggleDataset(\n        df = valid_df, \n        data_path = data_root,\n        augment = valid_augment,\n    )\n    valid_loader = DataLoader(\n        valid_dataset,\n        sampler     = SequentialSampler(valid_dataset),\n        batch_size  = 64,\n        drop_last   = False,\n        num_workers = 0,\n        pin_memory  = True,\n        collate_fn  = null_collate\n    )\n\n    assert(len(train_dataset)>=batch_size)\n    log.write('batch_size = %d\\n'%(batch_size))\n    log.write('train_dataset : \\n%s\\n'%(train_dataset))\n    log.write('valid_dataset : \\n%s\\n'%(valid_dataset))\n    log.write('\\n')\n\n    ## net ----------------------------------------\n    log.write('** net setting **\\n')\n    net = Net().cuda()\n    log.write('\\tinitial_checkpoint = %s\\n' % initial_checkpoint)\n\n    if initial_checkpoint is not None:\n        state_dict = torch.load(initial_checkpoint, map_location=lambda storage, loc: storage)\n        # for k in list(state_dict.keys()):\n        #      if any(s in k for s in ['logit',]): state_dict.pop(k, None)\n        # net.load_state_dict(state_dict,strict=False)\n\n        net.load_state_dict(state_dict,strict=True)  #True\n    else:\n        net.load_pretrain(pretrain_file = '../input/pytorch-pretrained-models/densenet121-a639ec97.pth', is_print=False)\n\n\n    log.write('net=%s\\n'%(type(net)))\n    log.write('\\n')\n\n\n\n    ## optimiser ----------------------------------\n    # if 0: ##freeze\n    #     for p in net.encoder1.parameters(): p.requires_grad = False\n    #     pass\n\n    #net.set_mode('train',is_freeze_bn=True)\n    #-----------------------------------------------\n\n    #optimizer = torch.optim.Adam(filter(lambda p: p.requires_grad, net.parameters()),lr=schduler(0))\n    #optimizer = torch.optim.RMSprop(net.parameters(), lr =0.0005, alpha = 0.95)\n    optimizer = torch.optim.SGD(filter(lambda p: p.requires_grad, net.parameters()), lr=schduler(0), momentum=0.0, weight_decay=0.0)\n\n    #num_iters   = 3000*1000 # use this for training longer\n    num_iters   = 1000  # comment this for training longer\n    iter_smooth = 50\n    iter_log    = 250\n    iter_valid  = 500\n    iter_save   = [0, num_iters-1]\\\n                   + list(range(0, num_iters, 1000))#1*1000\n\n    start_iter = 0\n    start_epoch= 0\n    rate       = 0\n    if initial_checkpoint is not None:\n        initial_optimizer = initial_checkpoint.replace('_model.pth','_optimizer.pth')\n        if os.path.exists(initial_optimizer):\n            checkpoint  = torch.load(initial_optimizer)\n            start_iter  = checkpoint['iter' ]\n            start_epoch = checkpoint['epoch']\n            #optimizer.load_state_dict(checkpoint['optimizer'])\n        pass\n\n    log.write('optimizer\\n  %s\\n'%(optimizer))\n    log.write('schduler\\n  %s\\n'%(schduler))\n    log.write('\\n')\n\n    ## start training here! ##############################################\n    log.write('** start training here! **\\n')\n    log.write('   batch_size=%d,  iter_accum=%d\\n'%(batch_size,iter_accum))\n    log.write('                    |----------------------- VALID------------------------------------|------- TRAIN/BATCH -----------\\n')\n    log.write('rate    iter  epoch | kaggle                    | loss               acc              | loss             | time       \\n')\n    log.write('----------------------------------------------------------------------------------------------------------------------\\n')\n              #0.01000  26.2  15.1 | 0.971 : 0.952 0.992 0.987 | 0.22, 0.07, 0.07 : 0.94, 0.98, 0.98 | 0.37, 0.13, 0.13 | 0 hr 13 min\n\n    def message(rate, iter, epoch, kaggle, valid_loss, train_loss, batch_loss, mode='print'):\n        if mode==('print'):\n            asterisk = ' '\n            loss = batch_loss\n        if mode==('log'):\n            asterisk = '*' if iter in iter_save else ' '\n            loss = train_loss\n\n        text = \\\n            '%0.5f %5.1f%s %4.1f | '%(rate, iter/1000, asterisk, epoch,) +\\\n            '%0.3f : %0.3f %0.3f %0.3f | '%(kaggle[1],*kaggle[0]) +\\\n            '%4.2f, %4.2f, %4.2f : %4.2f, %4.2f, %4.2f | '%(*valid_loss,) +\\\n            '%4.2f, %4.2f, %4.2f |'%(*loss,) +\\\n            '%s' % (time_to_str((timer() - start_timer),'min'))\n\n        return text\n\n    #----\n    kaggle = (0,0,0,0)\n    valid_loss = np.zeros(6,np.float32)\n    train_loss = np.zeros(3,np.float32)\n    batch_loss = np.zeros_like(train_loss)\n    iter = 0\n    i    = 0\n\n\n\n    start_timer = timer()\n    while  iter<num_iters:\n            \n        sum_train_loss = np.zeros_like(train_loss)\n        sum_train = np.zeros_like(train_loss)\n\n        optimizer.zero_grad()\n        for t, (input, truth, infor) in enumerate(train_loader):\n\n            batch_size = len(infor)\n            iter  = i + start_iter\n            epoch = (iter-start_iter)*batch_size/len(train_dataset) + start_epoch\n            \n            #if 0:\n            if (iter % iter_valid==0):\n                valid_loss, kaggle = do_valid(net, valid_loader, out_dir) #\n                pass\n\n            if (iter % iter_log==0):\n                print('\\r',end='',flush=True)\n                log.write(message(rate, iter, epoch, kaggle, valid_loss, train_loss, batch_loss, mode='log'))\n                log.write('\\n')\n\n            #if 0:\n            if iter in iter_save:\n                torch.save({\n                    #'optimizer': optimizer.state_dict(),\n                    'iter'     : iter,\n                    'epoch'    : epoch,\n                }, out_dir +'/checkpoint/%08d_optimizer.pth'%(iter))\n                if iter!=start_iter:\n                    torch.save(net.state_dict(),out_dir +'/checkpoint/%08d_model.pth'%(iter))\n                    pass\n\n            # learning rate schduler -------------\n            lr = schduler(iter)\n            if lr<0 : break\n            adjust_learning_rate(optimizer, lr)\n            rate = get_learning_rate(optimizer)\n\n            # one iteration update  -------------\n            #net.set_mode('train',is_freeze_bn=True)\n\n            net.train()\n            input = input.cuda()\n            truth = [t.cuda() for t in truth]\n\n            logit = data_parallel(net, input)\n            probability = logit_to_probability(logit)\n\n            loss = criterion(logit, truth)\n\n            (( 2*loss[0]+loss[1]+loss[2] )/iter_accum).backward()\n            if (iter % iter_accum)==0:\n                optimizer.step()\n                optimizer.zero_grad()\n\n            # print statistics  --------\n            loss = [l.item() for l in loss]\n            l = np.array([ *loss, ])*batch_size\n            n = np.array([ 1, 1, 1 ])*batch_size\n            batch_loss      = l/(n+1e-8)\n            sum_train_loss += l\n            sum_train      += n\n            if iter%iter_smooth == 0:\n                train_loss = sum_train_loss/(sum_train+1e-12)\n                sum_train_loss[...] = 0\n                sum_train[...]      = 0\n\n\n            print('\\r',end='',flush=True)\n            print(message(rate, iter, epoch, kaggle, valid_loss, train_loss, batch_loss, mode='print'), end='',flush=True)\n            i=i+1\n\n        pass  #-- end of one data loader --\n    pass #-- end of all iterations --\n    \n    log.write('\\n')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We will run the training but stop it, after 1000 iterations as the purpose of this kernel is to show you how to use Heng's code to train your models in kaggle kernels. You can train longer."},{"metadata":{"trusted":true},"cell_type":"code","source":"run_train()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Like I said before, this is a very strong baseline from Heng. It can be developed in many ways, some of which are:\n* Add mixup augmentation\n* Train with `serex50` model"},{"metadata":{},"cell_type":"markdown","source":"I hope this kernel was helpful to you in someways. I published this because I did not want Heng's Starter to be used by only those with compute power. Hopefully this kernel allows you to train your models using his codes using Kaggle GPUS, Google Colabs, and/or [Paperspace free GPUs](https://gradient.paperspace.com/free-gpu). You can follow this pipeline to train your models:\n> train in one kernel(for two hours) -> save model -> load model in another kernel -> train longer"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}