{"cells":[{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"'''\n!curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py\n!python pytorch-xla-env-setup.py --version nightly --apt-packages libomp5 libopenblas-dev\n!export XLA_USE_BF16=1\n'''","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install torch==1.7.0\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install torchvision ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install cloud-tpu-client==0.10 https://storage.googleapis.com/tpu-pytorch/wheels/torch_xla-1.7-cp37-cp37m-linux_x86_64.whl","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"thanks to @abhishek brother and @xhlulu as i borrowed a lot of code from them.have been learning a lot from them since last 1+ year. "},{"metadata":{},"cell_type":"markdown","source":"**pytorch is <3 **\n\ni will try different pretrained models in this kernel ,your suggestion in the comment box is highly appreciated, please check the #ChangeLog section of this kernel for getting version by version update,if you have any question about code please ask in the comment box."},{"metadata":{},"cell_type":"markdown","source":"NOTE : There could exist implementation bug,if you find any please let me know in the comment box so this solo worker can learn, thank you :)"},{"metadata":{},"cell_type":"markdown","source":"# ChangeLog\n* v1. Training for 7 epochs because i have ~6 hours gpu quota left now (model resnet101 of torchvision)\n* v2. MSELoss - 10 epoch (forcefully stopped training because of version 3 attempt)\n* v3. Adding sigmoid during inference\n* v4. inferencing by solving inference issue of v3\n* v5. trainset size = 59994 and model efficientnet-b3\n* v6. in version 5 i trained for 1 epoch and got ~0.6 lb score which took ~42 minutes for 1 epoch so now i will try for 8 epochs (overfitted)\n* v7. resnext50_32x4d with adamW\n* v8.  error\n* v9. trying to run On TPU\n* v10. trying to run On TPU\n* v10. trying again to run on TPU\n* v11. trying again to run on TPU for 1 epoch(hope this time it works)\n* v12. just training for 1 epoch\n* v13. Latest Updates on TPU --> 85-15 split, checking validation accuracy, model = efficientnet-b3, epoch = 10, OneCycleLearning rate, loss = BCEWithLogitsLoss\n\n* v14.  solving error of version 13\n* v15. i am not sure why it's taking a lot of time for training,i will try resnext50 for 1 epoch to see how long it takes\n* v16. 1 epoch took 20+ minutes so will try for 8 epoch now,i realized the validation accuracy calculation is slow,it is using 1 core i think,is there any way to use 8 core for validation accuracy calculation? please help in the comment box if you know how to speed up validation accuracy calculation, the training was fast but validation calculation step took some time,so i will try 95-5 split now\n* v17. couldn't finish training within 3 hours,will try 2 epochs now just to check how long it takes\n\n* v18. replacing avg_pool with AdaptiveAvgPool2d of efficientnetb-3,added logger and for checking the logger i will commit the kernel in debug mode for 5 epochs\n\n* v19. trying for 7 epoch and saving weights after 4th epoch\n* v20. in last version i had a bug,validation accuracy calculation was not correct and model was not  learning anything after  1 epoch,also the  validation loader was using only 1 core,i made it faster by  using validation sampler in validation loader and solved almost all the bugs that i was having previously after hours of debugging,let me know if i still have any bugs?\n* v21. 85-15 split,training for 10 epoch and added test set prediction using tpu\n* v22. i am not understanding why even in tpu 10 epochs taking more than 3 hours for effnet b3? also validation accuracy has some issue,i will try for 8 epoch  and steplr instead onecycle and 90-10 split"},{"metadata":{},"cell_type":"markdown","source":"# Import libraries and utility scripts"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"import os\n\nimport torch_xla\nimport torch_xla.debug.metrics as met\nimport torch_xla.distributed.data_parallel as dp\nimport torch_xla.distributed.parallel_loader as pl\nimport torch_xla.utils.utils as xu\nimport torch_xla.core.xla_model as xm\nimport torch_xla.distributed.xla_multiprocessing as xmp\nimport torch_xla.test.test_utils as test_utils\n\n\n\n\nimport torch\n\nimport warnings\n\nimport pandas as pd\nimport numpy as np\nimport torch.nn as nn\n\nfrom sklearn.model_selection import train_test_split\n\nfrom sklearn import metrics\nfrom transformers import AdamW\nfrom transformers import get_linear_schedule_with_warmup\n\nimport time\nimport torchvision\nimport torch.nn as nn\nfrom tqdm import tqdm_notebook as tqdm\n\nfrom PIL import Image, ImageFile\nfrom torch.utils.data import Dataset\nimport torch\nimport torch.optim as optim\nfrom torchvision import transforms\nfrom torch.optim import lr_scheduler\n\n\nimport sys\n\nimport gc\nimport os\nimport random\n\nimport skimage.io\nimport cv2\nfrom PIL import Image\nimport numpy as np\nimport pandas as pd\nimport scipy as sp\n\nimport sklearn.metrics\nfrom sklearn.metrics import accuracy_score\nfrom sklearn.model_selection import StratifiedKFold\n\nfrom functools import partial\n\nfrom torch.utils.data import DataLoader, Dataset\nimport torchvision.models as models\n\nfrom albumentations import Compose, Normalize, HorizontalFlip, VerticalFlip\nfrom albumentations.pytorch import ToTensorV2\n\n\nfrom contextlib import contextmanager\nfrom pathlib import Path\nfrom collections import defaultdict, Counter\n\nwarnings.filterwarnings(\"ignore\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#from : https://www.kaggle.com/yasufuminakama/panda-se-resnext50-classification-baseline/data\n# ====================================================\n# Utils\n# ====================================================\n\n@contextmanager\ndef timer(name):\n    t0 = time.time()\n    LOGGER.info(f'[{name}] start')\n    yield\n    LOGGER.info(f'[{name}] done in {time.time() - t0:.0f} s.')\n\n    \ndef init_logger(log_file='train.log'):\n    from logging import getLogger, DEBUG, FileHandler,  Formatter,  StreamHandler\n    \n    log_format = '%(asctime)s %(levelname)s %(message)s'\n    \n    stream_handler = StreamHandler()\n    stream_handler.setLevel(DEBUG)\n    stream_handler.setFormatter(Formatter(log_format))\n    \n    file_handler = FileHandler(log_file)\n    file_handler.setFormatter(Formatter(log_format))\n    \n    logger = getLogger('alaska2')\n    logger.setLevel(DEBUG)\n    logger.addHandler(stream_handler)\n    logger.addHandler(file_handler)\n    \n    return logger\n\nLOG_FILE = 'train.log'\nLOGGER = init_logger(LOG_FILE)\n\n\ndef seed_torch(seed=42):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.backends.cudnn.deterministic = True\n\nseed_torch(seed=42)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"BASE_PATH = \"/kaggle/input/alaska2-image-steganalysis\"\ntrain_imageids = pd.Series(os.listdir(BASE_PATH + '/Cover')).sort_values(ascending=True).reset_index(drop=True)\ntest_imageids = pd.Series(os.listdir(BASE_PATH + '/Test')).sort_values(ascending=True).reset_index(drop=True)\nsub = pd.read_csv('/kaggle/input/alaska2-image-steganalysis/sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#https://www.kaggle.com/xhlulu/alaska2-efficientnet-on-tpus\ndef append_path(pre):\n    return np.vectorize(lambda file: os.path.join(BASE_PATH, pre, file))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_filenames = np.array(os.listdir(\"/kaggle/input/alaska2-image-steganalysis/Cover/\"))\nlen(train_filenames)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#https://www.kaggle.com/xhlulu/alaska2-efficientnet-on-tpus\nnp.random.seed(0)\npositives = train_filenames.copy()\nnegatives = train_filenames.copy()\nnp.random.shuffle(positives)\nnp.random.shuffle(negatives)\n\njmipod = append_path('JMiPOD')(positives[:10000])\njuniward = append_path('JUNIWARD')(positives[10000:20000])\nuerd = append_path('UERD')(positives[20000:30000])\n\npos_paths = np.concatenate([jmipod, juniward, uerd])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#https://www.kaggle.com/xhlulu/alaska2-efficientnet-on-tpus\ntest_paths = append_path('Test')(sub.Id.values)\nneg_paths = append_path('Cover')(negatives[:30000])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_paths = np.concatenate([pos_paths, neg_paths])\ntrain_labels = np.array([1] * len(pos_paths) + [0] * len(neg_paths))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#https://www.kaggle.com/xhlulu/alaska2-efficientnet-on-tpus\ntrain_paths, valid_paths, train_labels, valid_labels = train_test_split(\n    train_paths, train_labels, test_size=0.005, random_state=2020)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"len(valid_labels)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"l=np.array([train_paths,train_labels])\ntraindataset = pd.DataFrame({ 'images': list(train_paths), 'label': train_labels},columns=['images','label'])\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"val_l=np.array([valid_paths,valid_labels])\nvaliddataset = pd.DataFrame({ 'images': list(valid_paths), 'label': valid_labels},columns=['images','label'])\nvaliddataset.head(3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"len(traindataset)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"traindataset.head(2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"#i use this line of code for debugging\n'''traindataset = traindataset.head(5000)\nvaliddataset = validdataset.head(200) '''\nlen(traindataset)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"len(validdataset)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"image = Image.open(train_paths[50] )\nimage","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class train_images(Dataset):\n\n    def __init__(self, csv_file):\n\n        self.data = csv_file\n\n    def __len__(self):\n        return len(self.data)\n\n    def __getitem__(self, idx):\n        #print(idx)\n        img_name =  self.data.loc[idx][0]\n        image = Image.open(img_name)\n        image = image.resize((512, 512), resample=Image.BILINEAR)\n        label = self.data.loc[idx][1] #torch.tensor(self.data.loc[idx, 'label'])\n        return {'image': transforms.ToTensor()(image),\n                'label': label\n                }","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_dataset = train_images(traindataset)\nvalid_dataset = train_images(validdataset)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install efficientnet_pytorch\nfrom efficientnet_pytorch import EfficientNet\ncriterion = torch.nn.BCEWithLogitsLoss() # \nnum_epochs = 10\nNUM_EPOCH = num_epochs\nfrom torch.optim.lr_scheduler import OneCycleLR\n\nBATCH_SIZE = 12\n#model = torchvision.models.resnext50_32x4d(pretrained=True)\n#model.load_state_dict(torch.load(\"../input/pytorch-pretrained-models/resnet101-5d3b4d8f.pth\"))\nmodel = EfficientNet.from_name('efficientnet-b3')\n\n#model.avg_pool = nn.AdaptiveAvgPool2d(1)\nnum_ftrs = model._fc.in_features\nmodel._fc = nn.Linear(num_ftrs, 1)\n#model.load_state_dict(torch.load(\"../input/pytorch-transfer-learning-baseline/model.bin\"))\n#device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# https://www.kaggle.com/anokas/weighted-auc-metric-updated\n\ndef alaska_weighted_auc(y_true, y_valid):\n    tpr_thresholds = [0.0, 0.4, 1.0]\n    weights = [2,   1]\n\n    fpr, tpr, thresholds = metrics.roc_curve(y_true, y_valid, pos_label=1)\n\n    # size of subsets\n    areas = np.array(tpr_thresholds[1:]) - np.array(tpr_thresholds[:-1])\n\n    # The total area is normalized by the sum of weights such that the final weighted AUC is between 0 and 1.\n    normalization = np.dot(areas, weights)\n\n    competition_metric = 0\n    for idx, weight in enumerate(weights):\n        y_min = tpr_thresholds[idx]\n        y_max = tpr_thresholds[idx + 1]\n        mask = (y_min < tpr) & (tpr < y_max)\n        # pdb.set_trace()\n\n        x_padding = np.linspace(fpr[mask][-1], 1, 100)\n\n        x = np.concatenate([fpr[mask], x_padding])\n        y = np.concatenate([tpr[mask], [y_max] * len(x_padding)])\n        y = y - y_min  # normalize such that curve starts at y=0\n        score = metrics.auc(x, y)\n        submetric = score * weight\n        best_subscore = (y_max - y_min) * weight\n        competition_metric += submetric\n\n    return competition_metric / normalization","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#https://www.kaggle.com/dhananjay3/pytorch-xla-for-tpu-with-multiprocessing\n#https://www.kaggle.com/abhishek/very-simple-pytorch-training-0-59/data\n\ndef train_model():\n    global train_dataset, valid_dataset\n    \n    torch.manual_seed(42)\n    \n    train_sampler = torch.utils.data.distributed.DistributedSampler(\n        train_dataset,\n        num_replicas=xm.xrt_world_size(),\n        rank=xm.get_ordinal(),\n        shuffle=True)\n    \n    train_loader = torch.utils.data.DataLoader(\n        train_dataset,\n        batch_size=BATCH_SIZE,\n        sampler=train_sampler,\n        num_workers=0,\n        drop_last=True) # print(len(train_loader))\n    \n    valid_sampler = torch.utils.data.distributed.DistributedSampler(\n        valid_dataset,\n        num_replicas=xm.xrt_world_size(),\n        rank=xm.get_ordinal(),\n        )\n        \n    valid_loader = torch.utils.data.DataLoader(\n        valid_dataset,\n        batch_size=BATCH_SIZE ,\n        sampler=valid_sampler,\n        shuffle=False,\n        num_workers=0,\n        drop_last=True)\n    \n    #xm.master_print(f\"Train for {len(train_loader)} steps per epoch\")\n    LOGGER.debug(f\"Train for {len(train_loader)} steps per epoch\")\n    # Scale learning rate to num cores\n    lr  = 0.001 * xm.xrt_world_size()\n\n    # Get loss function, optimizer, and model\n    device = xm.xla_device()\n\n    #model = model()\n    '''\n    for param in model.base_model.parameters(): # freeze some layers\n        param.requires_grad = False'''\n    \n    \n    global model\n    \n    model = model.to(device)\n\n    criterion = torch.nn.MSELoss() #  BCEWithLogitsLoss\n    #criterion = torch.nn.CrossEntropyLoss()\n    optimizer = optim.AdamW(model.parameters(), lr=lr)\n    scheduler = lr_scheduler.StepLR(optimizer, step_size=10)\n    \n    \n\n    \n    def train_loop_fn(loader):\n        tracker = xm.RateTracker()\n        model.train()\n        #xm.master_print('Epoch {}/{}'.format(epoch, num_epochs - 1))\n        LOGGER.debug('Epoch {}/{}'.format(epoch, num_epochs - 1))\n        #xm.master_print('-' * 10)\n\n        LOGGER.debug('-' * 10)\n        scheduler.step()\n        \n        running_loss = 0.0\n        tk0 = tqdm(loader, total=int(len(train_loader)))\n        counter = 0\n        for bi, d in enumerate(tk0):\n            inputs = d[\"image\"]\n            labels = d[\"label\"].view(-1, 1)\n            inputs = inputs.to(device, dtype=torch.float)\n            labels = labels.to(device, dtype=torch.float)\n            #labels = labels.to(device, dtype=torch.long)\n            optimizer.zero_grad()\n            #with torch.set_grad_enabled(True):\n            outputs = model(inputs)\n            loss = criterion(outputs, labels)\n            #loss = criterion(outputs, torch.max(labels, 1)[1])\n            loss.backward()\n            xm.optimizer_step(optimizer)\n            running_loss += loss.item() * inputs.size(0)\n            #print(running_loss)\n            counter += 1\n            tk0.set_postfix(loss=(running_loss / (counter * BATCH_SIZE)))\n        epoch_loss = running_loss / len(train_loader)\n        \n        #xm.master_print('Training Loss: {:.8f}'.format(epoch_loss))\n        LOGGER.debug('Training Loss: {:.8f}'.format(epoch_loss))\n\n                \n    def test_loop_fn(loader):\n        \n        tk0 = tqdm(loader, total=int(len(valid_loader)))\n        counter = 0\n        total_samples, correct = 0, 0\n        for bi, d in enumerate(tk0):\n            inputs = d[\"image\"]\n            labels = d[\"label\"].view(-1, 1)\n            inputs = inputs.to(device, dtype=torch.float)\n            labels = labels.to(device, dtype=torch.float)\n            #labels = labels.to(device, dtype=torch.long)\n            #optimizer.zero_grad()\n            \n            #with torch.no_grad():\n                \n            output = model(inputs)\n                \n            pred = output.max(1, keepdim=True)[1]\n            correct += pred.eq(labels.view_as(pred)).sum().item()\n            total_samples += inputs.size()[0]\n        accuracy = 100.0 * correct / total_samples\n        \n        \n        #auc_score = alaska_weighted_auc(labels.cpu().numpy(), output.cpu().numpy())\n        #LOGGER.debug(\"auc_score according to competition metric = {} \".format(auc_score))\n        #print('[xla:{}] Accuracy={:.4f}%'.format(xm.get_ordinal(), accuracy), flush=True)\n        model.train()\n        return accuracy\n\n    # Train - valid  loop\n    accuracy = []\n    for epoch in range(1, num_epochs + 1):\n        start = time.time()\n        para_loader = pl.ParallelLoader(train_loader, [device])\n        train_loop_fn(para_loader.per_device_loader(device))\n        \n        para_loader = pl.ParallelLoader(valid_loader, [device])\n        accuracy.append(test_loop_fn(para_loader.per_device_loader(device)))\n        #xm.master_print(\"Finished training epoch {}  Val-Acc {:.4f} in {:.4f} sec\".format(epoch, accuracy[-1],   time.time() - start))        \n        \n        \n        LOGGER.debug(\"Finished training epoch {}  Val-Acc {:.4f} in {:.4f} sec\".format(epoch, accuracy[-1],   time.time() - start))   \n        valauc = accuracy[-1]\n        if(epoch>9):\n            xm.save(model.state_dict(), f\"./epoch{epoch}valauc{valauc}.bin\")\n    return accuracy","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Begin Training "},{"metadata":{},"cell_type":"markdown","source":"**output of the  cell  below is disabled,for checking full train log please check the train.log file attached as kernels output**"},{"metadata":{"trusted":true,"_kg_hide-output":true,"_kg_hide-input":true},"cell_type":"code","source":"# Start training processes\n\ndef _mp_fn(rank, flags):\n    global acc_list\n    torch.set_default_tensor_type('torch.FloatTensor')\n    res = train_model()\n\nFLAGS={}\nxmp.spawn(_mp_fn, args=(FLAGS,), nprocs=8, start_method='fork')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"xm.get_xla_supported_devices()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Inference"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"class test_images(Dataset):\n\n    def __init__(self, csv_file):\n\n        self.data = csv_file\n\n    def __len__(self):\n        return len(self.data)\n\n    def __getitem__(self, idx):\n        img_name =  self.data.loc[idx][0]\n        image = Image.open(img_name)\n        image = image.resize((512, 512), resample=Image.BILINEAR)\n        #label = self.data.loc[idx][1] #torch.tensor(self.data.loc[idx, 'label'])\n        #image = self.transform(image)\n        return {'image': transforms.ToTensor()(image)}\n\ntestdataset = pd.DataFrame({ 'images': list(test_paths)},columns=['images'])\n#testdataset.head(2)\ntestdataset = test_images(testdataset)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true,"_kg_hide-input":true},"cell_type":"code","source":"Test_BATCH_SIZE = 1    \ntest_sampler = torch.utils.data.distributed.DistributedSampler(\n      testdataset,\n      num_replicas=xm.xrt_world_size(),\n      rank=xm.get_ordinal()\n      )\n\ntest_data_loader = torch.utils.data.DataLoader(\n    testdataset,\n    batch_size=Test_BATCH_SIZE,\n    #sampler=test_sampler,\n    #drop_last=True,\n    num_workers=0\n)\n\ndevice = xm.xla_device()\n\nmodel = model.to(device)\n\ntestpara_loader = pl.ParallelLoader(test_data_loader, [device])\n\nsub[\"Label\"] = pd.to_numeric(sub[\"Label\"].astype(float))\n","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"%%time\n#test_loader = torch.utils.data.DataLoader(testdataset, batch_size=1, shuffle=False) # test_set contains only images directory\n\n\ntest_loader =  testpara_loader.per_device_loader(device)\n\n\n#print(type(test_loader))\n'''\nfor param in model.parameters():\n    param.requires_grad = False '''\n\nprediction_list = []\ntk0 = tqdm(test_loader)\n\nfor i, x_batch in enumerate(tk0):\n    \n    x_batch = x_batch[\"image\"].to(device)\n    \n    #x_batch.to(device)\n    #pred =  model(x_batch)\n    pred =  model(x_batch.to(device))\n    #prediction_list.append(pred.cpu())\n    #print( type(pred.item()))\n    #print(\"\\n\")\n    sub.Label[i] = pred.item()\n    #print(sub.Label[i])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"sub.to_csv('submission.csv', index=False)\nsub.head(5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# TakeAways : \n* [very simple pytorch training](https://www.kaggle.com/abhishek/very-simple-pytorch-training-0-59/data)\n\n* [Alaska2: EfficientNet on TPUs](https://www.kaggle.com/xhlulu/alaska2-efficientnet-on-tpus)\n\n* [bert multi lingual tpu training (8 cores)](https://www.kaggle.com/abhishek/bert-multi-lingual-tpu-training-8-cores)\n* [Pytorch/XLA for TPU with multiprocessing](https://www.kaggle.com/dhananjay3/pytorch-xla-for-tpu-with-multiprocessing)\n\n* [PANDA / se_resnext50 classification baseline](https://www.kaggle.com/yasufuminakama/panda-se-resnext50-classification-baseline/data)\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}