{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# **EfficientDet Pytorch Starter**","metadata":{}},{"cell_type":"markdown","source":"Hi Everyone,\n\nI couldnt see an existing one around, so thought I would create a quick starter for an efficientdet pipeline using pytorch. This kernel uses the package by Ross Wightman available at github [here](https://github.com/rwightman/efficientdet-pytorch) (albeit a rather old version for compatibility)\n\nThanks to Alex Shonenkov [here](https://www.kaggle.com/shonenkov/training-efficientdet), as this is losely based on his starter for the GWD competition. I had to make some changes to the timm-efficiendet package to get it to work correctly.\n\nFor the people who haven't seen it before, EfficientDet is a one shot object detection model originally published by Mingxing Tan, Ruoming Pang, Quoc V. Le from Google Research. The paper is available [here](https://arxiv.org/abs/1911.09070). It's performance compared to other current object detection models is shown below:","metadata":{}},{"cell_type":"markdown","source":"![image.png](https://github.com/google/automl/blob/master/efficientdet/g3doc/flops.png?raw=true)\n\n[Source](https://github.com/google/automl/blob/master/efficientdet/g3doc/flops.png)","metadata":{}},{"cell_type":"markdown","source":"Right, let's start the code. Lets do our installs:","metadata":{}},{"cell_type":"code","source":"#Need to install these - Only in the submission version for DICOM processing. Not needed here\n#!conda install -c conda-forge pillow -y\n#!conda install -c conda-forge pydicom -y\n#!conda install -c conda-forge gdcm -y\n#!pip install pylibjpeg pylibjpeg-libjpeg\n!pip install --no-deps '../input/timm-package/timm-0.1.26-py3-none-any.whl' > /dev/null\n!pip install --no-deps '../input/pycocotools/pycocotools-2.0-cp37-cp37m-linux_x86_64.whl' > /dev/null","metadata":{"_uuid":"5bf4f812-9357-4de0-905a-8d3b6e2eae2f","_cell_guid":"9622e339-344d-44a7-a3c5-92b9b2041b4a","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-07-07T01:02:45.477508Z","iopub.execute_input":"2021-07-07T01:02:45.477886Z","iopub.status.idle":"2021-07-07T01:02:49.840585Z","shell.execute_reply.started":"2021-07-07T01:02:45.477809Z","shell.execute_reply":"2021-07-07T01:02:49.839418Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we build our training bench for EfficientDet B1 (or just replace with whichever EfficientDet level you wish to use). Thanks to user @mathurinache for the pretrained weights [here](https://www.kaggle.com/mathurinache/efficientdet).","metadata":{}},{"cell_type":"code","source":"import sys\nsys.path.insert(0, \"../input/timmefficienctdetpytorchstable/archive\")\nsys.path.insert(0, \"../input/omegaconf\")\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)  \nimport torch\nimport torch.nn.functional as F\nimport torchvision\nimport traceback\nfrom tqdm import tqdm\nfrom torch.utils.data import Dataset\nimport torchvision.models as models\nfrom PIL import Image\nfrom torch import optim\nfrom effdet import *\nfrom effdet.efficientdet import HeadNet\nfrom effdet.anchors import Anchors, AnchorLabeler, generate_detections\nfrom effdet.loss import DetectionLoss\n\nclass EfficientDetTrainer(torch.nn.Module):\n    \n    def __init__(self, model, config, device):\n        \n        super(EfficientDetTrainer, self).__init__()\n        self.model = model\n        self.config = config\n        self.my_anchors = Anchors(\n                config.min_level, config.max_level,\n                config.num_scales, config.aspect_ratios,\n                config.anchor_scale, config.image_size, device)\n        self.a = AnchorLabeler(self.my_anchors, 1, match_threshold=0.5)\n        self.loss_fn = DetectionLoss(config)     \n            \n    def forward(self, x, boxes, classes):\n        \n        class_out, box_out = self.model(x)\n        cls_targets = []\n        box_targets = []\n        num_positives = []\n        \n        for i in range(inputs.shape[0]):\n            gt_class_out, gt_box_out, num_positive = self.a.label_anchors(boxes[i], classes[i])\n            cls_targets.append(gt_class_out)\n            box_targets.append(gt_box_out)\n            num_positives.append(num_positive)\n                \n        loss, class_loss, box_loss = self.loss_fn(class_out, box_out, cls_targets, box_targets, num_positives)\n        return loss\n\ndef get_efficientDet():\n    \n    config = get_efficientdet_config('tf_efficientdet_d1')\n    net = EfficientDet(config, pretrained_backbone=False)\n    config.num_classes = 1\n    config.image_size = 512\n    net.class_net = HeadNet(config, num_outputs=config.num_classes, norm_kwargs=dict(eps=.001, momentum=.01))\n    return net, config\n\nmodel, config = get_efficientDet()\n\n","metadata":{"_uuid":"46130165-c344-420d-a752-e43631e8bd67","_cell_guid":"e011fd52-5cf9-458c-a9e9-43b115ec5b96","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-07-07T01:02:57.037201Z","iopub.execute_input":"2021-07-07T01:02:57.037538Z","iopub.status.idle":"2021-07-07T01:02:59.276871Z","shell.execute_reply.started":"2021-07-07T01:02:57.037508Z","shell.execute_reply":"2021-07-07T01:02:59.276132Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Next, we create our dataset class and our dataloaders. Thanks to xhulu [here](https://www.kaggle.com/xhlulu/siim-covid-19-convert-to-jpg-256px) for the converted images (DICOM to jpg and re-sized to 512px). I have only used images with bounding boxes for training (i.e. no background only classes). I suspect this may not be a good strategy and needs to be investigated......","metadata":{}},{"cell_type":"code","source":"#Here we are getting only the images with bounding boxes for training\nerror_list = ['none 1 0 0 1 1']\nimport os\nfiles = []\nfor dirname, _, filenames in os.walk('/kaggle/input/siim-covid19-detection/train/'):\n    for filename in filenames:\n        files.append(os.path.join(dirname, filename))\n        \nNormalizer = torchvision.transforms.Compose([torchvision.transforms.ToTensor(),\n                                             torchvision.transforms.Normalize(mean=[0.485, 0.456, 0.406],\n                                             std=[0.229, 0.224, 0.225])])\n\n\ncsv_image_trainer = pd.read_csv(\"/kaggle/input/siim-covid19-detection/train_image_level.csv\")\ncsv_study_trainer = pd.read_csv(\"/kaggle/input/siim-covid19-detection/train_study_level.csv\")\nmeta_csv = pd.read_csv('/kaggle/input/resize512px/meta.csv')\n\n\nlocalizer_files = []\n#Remove all the ones with only background for training regression model\nfor file in files:\n    file = file.split('/')\n    image_file = file[7].split('.')[0] + '_image'\n    image_row = csv_image_trainer.loc[csv_image_trainer['id'] == image_file]\n    a = image_row['label'].to_list()\n    if a != error_list:\n        localizer_files.append(file)\n    \n    \nprint(len(files))\nprint(len(localizer_files))\n\n# ======================\n# ======================\n# Params\nBATCH_SIZE = 2\nN_WORKERS = 4\nN_EPOCHS = 3 # You will obviously want to train it for more....\n\ndef collate_fn(batch):\n    return tuple(zip(*batch))\n\nclass COVIDXRay_Dataset(Dataset):\n\n    def __init__(self, dcm_file_list, transform, csv_image_trainer, csv_study_trainer, meta_csv):\n\n        self.dcm_file_list = dcm_file_list\n        self.transform = transform\n        self.csv_image_trainer = csv_image_trainer\n        self.csv_study_trainer = csv_study_trainer\n        self.meta_csv = meta_csv\n\n    def __len__(self):\n        \n        return len(self.dcm_file_list)\n\n    def __getitem__(self, idx):\n        \n        file = self.dcm_file_list[idx]\n        #Now our image level ground truth\n        image_file = file[7].split('.')[0] + '_image'\n        image_dir = '/kaggle/input/resize512px/train/' + file[7].split('.')[0] + '.jpg'\n        img =  Image.open(image_dir).convert('RGB')\n        \n        #Get from Xhulu's Notebook\n        metadata_record = self.meta_csv.loc[self.meta_csv['image_id'] == file[7].split('.')[0]]\n        #Reversed from what we are expecting....\n        width = metadata_record['dim1']\n        height = metadata_record['dim0']\n        w_factor = 512/width\n        h_factor = 512/height\n        img = self.transform(img)\n        \n        #First of all we get our study level ground truth\n        study_file = file[5] + '_study'\n        study_row = csv_study_trainer.loc[csv_study_trainer['id'] == study_file]\n        study_tgt = study_row.values[0][1:5].astype(np.float)\n        \n        #Now our image level ground truth\n        image_file = file[7].split('.')[0] + '_image'\n        image_row = csv_image_trainer.loc[csv_image_trainer['id'] == image_file]\n\n        #Parse the boxes\n        in_boxes = str(image_row['label'].values[0]).split(' ')\n        q = int(len(in_boxes)/6)\n        boxes = np.zeros((q,4))\n        classes = np.ones((q))\n        #Not sure this is the best way to do this\n        for i in range(0,q):\n            offset = i * 6\n            #boxes[i,0] = float(in_boxes[2+offset]) * w_factor #x1\n            #boxes[i,1] = float(in_boxes[3+offset]) * h_factor #y1\n            #boxes[i,2] = float(in_boxes[4+offset]) * w_factor #x2\n            #boxes[i,3] = float(in_boxes[5+offset]) * h_factor #y2\n            #yxyx format here due to original tensorflow implementation            \n            boxes[i,1] = float(in_boxes[2+offset]) * w_factor #y1\n            boxes[i,0] = float(in_boxes[3+offset]) * h_factor #x1\n            boxes[i,3] = float(in_boxes[4+offset]) * w_factor #y2\n            boxes[i,2] = float(in_boxes[5+offset]) * h_factor #x2\n            \n        target = {\"bbox\":torch.Tensor(boxes),\"cls\":torch.Tensor(classes)}        \n        study_tgt = torch.Tensor(study_tgt)\n        return img, target, study_tgt\n    \n    \ntrain_dataset = COVIDXRay_Dataset(\n        localizer_files, Normalizer, csv_image_trainer, csv_study_trainer, meta_csv\n)\n    \n\noverall_length = len(train_dataset)\nvalidation_length = int(len(train_dataset) * 0.2) + 1\ntrain_length = int(len(train_dataset) * 0.8)\n\ntrain_dataset, validation_dataset = torch.utils.data.random_split(train_dataset, [train_length, validation_length])\n\ndata_loader_train = torch.utils.data.DataLoader(\n    train_dataset,\n    batch_size=BATCH_SIZE,\n    num_workers=N_WORKERS,\n    shuffle=True,\n    collate_fn=collate_fn\n)\n\ndata_loader_test = torch.utils.data.DataLoader(\n    validation_dataset,\n    batch_size=BATCH_SIZE,\n    num_workers=N_WORKERS,\n    shuffle=True,\n    collate_fn=collate_fn\n)\n\n","metadata":{"execution":{"iopub.status.busy":"2021-07-07T01:03:14.174978Z","iopub.execute_input":"2021-07-07T01:03:14.17535Z","iopub.status.idle":"2021-07-07T01:03:47.63253Z","shell.execute_reply.started":"2021-07-07T01:03:14.17532Z","shell.execute_reply":"2021-07-07T01:03:47.63161Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Ok, we are nearly ready to go. We create our instance for the trainer and add any callbacks we need. Then we start our training/validation loop for the required number of epochs. We save our weights file when the testing loss decreases.","metadata":{}},{"cell_type":"code","source":"device = torch.device(\"cuda:0\")\n#Move it to the device\nmodel = model.to(device)\n\n#Optimizer\nmy_trainer = EfficientDetTrainer(model, config, device)\noptimizer = optim.Adam(model.parameters(), lr=.00005)\nscheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='min', factor=0.1, \n                                           patience=3, threshold=0.0001, threshold_mode='rel', \n                                           cooldown=0, min_lr=0, eps=1e-08, verbose=False)\nbest_loss = 99999\n\ntry:\n    for epoch in range(N_EPOCHS):\n\n        print('Epoch {}/{}'.format(epoch, N_EPOCHS - 1))\n        print('-' * 10)\n\n        model.train()\n        tr_loss = 0\n        tst_loss = 0\n\n        tk0 = tqdm(data_loader_train, desc=\"Iteration\")\n        \n        for batch_idx, (inputs, target, study_tgt) in enumerate(tk0):\n            \n            boxes = [torch.Tensor(t['bbox']).to(device).float() for t in target]\n            classes = [torch.Tensor(t['cls']).to(device).float() for t in target]\n            inputs = torch.stack(inputs)\n            inputs = inputs.to(device).float()\n            loss = my_trainer(inputs,boxes,classes)\n            loss.backward()\n            tr_loss += loss.item()\n            optimizer.step()\n            optimizer.zero_grad()\n              \n        epoch_loss = tr_loss / len(data_loader_train)\n        \n        print('Training Localization Loss: {:.4f}'.format(epoch_loss))\n        #Leave in training mode for now - unless we need to make the switch for Evaluation mode\n        #model.eval()\n\n        for batch_idx, (inputs, target, study_tgt) in enumerate(data_loader_test):\n            \n            boxes = [torch.Tensor(t['bbox']).to(device).float() for t in target]\n            classes = [torch.Tensor(t['cls']).to(device).float() for t in target]\n            inputs = torch.stack(inputs)\n            inputs = inputs.to(device).float()\n            loss = my_trainer(inputs,boxes,classes)\n            tst_loss += loss.item()\n            study_tgt = [t.to(device) for t in study_tgt]\n            \n        epoch_loss = tst_loss / len(data_loader_test)\n        scheduler.step(epoch_loss)\n        \n        print('Testing Loss: {:.4f}'.format(epoch_loss))\n        if epoch_loss < best_loss:\n            best_loss = epoch_loss\n            model.eval()\n            torch.save(model.state_dict(), './weights.pth')\n            print('Test Loss Improved....Saving Model')\n            \nexcept:\n    traceback.print_exc(file=sys.stdout)","metadata":{"execution":{"iopub.status.busy":"2021-07-07T01:03:55.624346Z","iopub.execute_input":"2021-07-07T01:03:55.624656Z","iopub.status.idle":"2021-07-07T01:32:25.338583Z","shell.execute_reply.started":"2021-07-07T01:03:55.624628Z","shell.execute_reply":"2021-07-07T01:32:25.337515Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Thanks everyone for reading. I hope it's useful. Let me know if any of the steps need to be described in more detail.","metadata":{}}]}