{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"I normally tend to use Pytorch as my framework but I wanted to try Pytorch Lightning after I heard about it. \nPytorch Lightning helps with structured Pytorch Code and scalability on Multi GPU environment.\n\nPytorch lighntning is designed to help you easily follow a pytorch based training loop and ease modifications that you may want. Want to use a new scheduler ? Then simply modify the configure_optimizer method ! The beauty of it is that it automates all the boring stuff that clogs a pure pytorch code. All these loops, .zero_grad(), .eval(), torch.save etc. are gone and handled by the framework. You just have to focus on the ML part of it. The best things for researchers is that it comes with automated logs through tensorboard to compare your many experiments and easy switches between GPU, DataParallel, TPU mixed precision etc.","metadata":{}},{"cell_type":"markdown","source":"Torch Metrics along with lightning has made metric handling way easier, with a simple intialisation of metric and variable reference makes the task of updating way easier. <BR>\nThe concept of Log helps to keep track of metrics easily as a progressbar or as a Logger like Tensorboard.","metadata":{}},{"cell_type":"markdown","source":"The whole kernel was made possible using the following references:\n* Lightning Docs: [Lighting Docs](https://pytorch-lightning.readthedocs.io/en/latest/)\n* Lightning based Kernel on Kaggle: [Arnaud's Kernel](https://www.kaggle.com/arroqc/siim-isic-pytorch-lightning-starter-seresnext50/notebook)\n* Preprocessing of DataFrame: [Tanay's Kernel](https://www.kaggle.com/heyytanay/siim-pytorch-classification-only-training-effnets)\n* Dicom to image conversion : [Xhlulu's Kernel](https://www.kaggle.com/xhlulu/siim-covid-19-convert-to-jpg-256px)\n* Logic behind the metric from: [This Thread](https://www.kaggle.com/c/siim-covid19-detection/discussion/241300)\n","metadata":{}},{"cell_type":"markdown","source":"# Installing Dependencies and Libraries","metadata":{}},{"cell_type":"code","source":"!cp /kaggle/input/gdcm-conda-install/gdcm.tar .\n!tar -xvzf gdcm.tar\n!conda install --offline ./gdcm/gdcm-2.8.9-py37h71b2a6d_0.tar.bz2\n!rm -rf ./gdcm.tar","metadata":{"execution":{"iopub.status.busy":"2021-07-04T17:20:13.035457Z","iopub.execute_input":"2021-07-04T17:20:13.035817Z","iopub.status.idle":"2021-07-04T17:20:39.695040Z","shell.execute_reply.started":"2021-07-04T17:20:13.035739Z","shell.execute_reply":"2021-07-04T17:20:39.694037Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np \nimport pandas as pd \nimport os\nimport cv2\nfrom tqdm import tqdm \nimport glob\nimport sys\n\nimport torch \nimport torch.nn as nn \nimport torch.nn.functional as F\nfrom torch.utils.data import Dataset, DataLoader\nimport albumentations as A\nfrom albumentations.pytorch import ToTensorV2\n\nimport pytorch_lightning as pl\nimport torchmetrics\nfrom pytorch_lightning import loggers as pl_loggers\n\n\nsys.path.append(\"../input/timm-pytorch-image-models/pytorch-image-models-master\")\nimport timm\n\nsys.path.append(\"../input/iterative-stratification/iterative-stratification-master\")\nfrom iterstrat.ml_stratifiers import MultilabelStratifiedKFold\n\nimport pydicom\nfrom pydicom.pixel_data_handlers.util import apply_voi_lut\nfrom sklearn.metrics import roc_auc_score","metadata":{"_uuid":"df6345bd-0b4d-4a8b-aafa-5dd87f85b20d","_cell_guid":"4dc476cc-cc59-4f23-870d-a8fab851e73b","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-07-04T17:20:39.698632Z","iopub.execute_input":"2021-07-04T17:20:39.698907Z","iopub.status.idle":"2021-07-04T17:20:44.618535Z","shell.execute_reply.started":"2021-07-04T17:20:39.698879Z","shell.execute_reply":"2021-07-04T17:20:44.617548Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Configurations ","metadata":{}},{"cell_type":"code","source":"TRAIN_BATCH_SIZE = 4\nVAL_BATCH_SIZE = 8\nIMG_SIZE = 512\nLABELS = ['Negative for Pneumonia','Typical Appearance', 'Indeterminate Appearance', 'Atypical Appearance']\nNUM_FOLDS = 6\nMODEL_NAME = 'efficientnet_b7'\nFOLD = 3\nEPOCHS = 3\nSTUDY_LEN = 0","metadata":{"execution":{"iopub.status.busy":"2021-07-04T17:20:44.620611Z","iopub.execute_input":"2021-07-04T17:20:44.620999Z","iopub.status.idle":"2021-07-04T17:20:44.627380Z","shell.execute_reply.started":"2021-07-04T17:20:44.620956Z","shell.execute_reply":"2021-07-04T17:20:44.625926Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This function preprocesses the csv file with the labels provided \n* The function takes train as an argument. When true indicates the data to be processed is for train else to be processed for test\n* When handling test data the sample_predictions dataframe has a mix of classification and detection records. Therefore the records are segregated using a map function\n\nThe training part of the function takes reference from: [Tanay's Kernel](https://www.kaggle.com/heyytanay/siim-pytorch-classification-only-training-effnets)","metadata":{}},{"cell_type":"code","source":"def preprocess_df(train = True):\n    if train:\n        df_image = pd.read_csv(\"../input/siim-covid19-detection/train_study_level.csv\")\n        df_det = pd.read_csv(\"../input/siim-covid19-detection/train_image_level.csv\")\n        df_image['StudyInstanceUID'] = df_image['id'].apply(lambda x : x[:-6])\n        df = df_det.merge(df_image, on='StudyInstanceUID')\n        path = []\n        TRAIN_DIR = \"../input/siim-covid19-detection/train/\"\n        for instance_id in tqdm(df['StudyInstanceUID']):\n            path.append(glob.glob(os.path.join(TRAIN_DIR, instance_id +\"/*/*\"))[0])\n        df['path'] = path\n        df = df.drop(['id_x', 'id_y'], axis=1)\n        return df\n    \n#     else:\n#         df= pd.read_csv(\"../input/siim-covid19-detection/sample_submission.csv\")\n#         study_indices = df['id'].apply(lambda x : x[-6:])\n#         STUDY_LEN =0\n#         for i in range(len(study_indices)):\n#             if study_indices[i] == '_image':\n#                 STUDY_LEN = i\n#                 break\n            \n#         df['StudyInstanceUID'] = df['id'].apply(lambda x : x[:-6])\n#         df = df.iloc[:STUDY_LEN,:]\n#         path = []\n#         TEST_DIR = \"../input/siim-covid19-detection/test\"\n#         for instance_id in tqdm(df['StudyInstanceUID']):\n#             path.append(glob.glob(os.path.join(TEST_DIR, instance_id +\"/*/*\"))[0])\n#         df['path'] = path\n\n#         return df,STUDY_LEN\n    ","metadata":{"execution":{"iopub.status.busy":"2021-07-04T17:20:44.629320Z","iopub.execute_input":"2021-07-04T17:20:44.629685Z","iopub.status.idle":"2021-07-04T17:20:44.639159Z","shell.execute_reply.started":"2021-07-04T17:20:44.629649Z","shell.execute_reply":"2021-07-04T17:20:44.638273Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def dicom2array(path, voi_lut=True, fix_monochrome=True):\n    dicom = pydicom.read_file(path)\n    # VOI LUT (if available by DICOM device) is used to\n    # transform raw DICOM data to \"human-friendly\" view\n    if voi_lut:\n        data = apply_voi_lut(dicom.pixel_array, dicom)\n    else:\n        data = dicom.pixel_array\n    # depending on this value, X-ray may look inverted - fix that:\n    if fix_monochrome and dicom.PhotometricInterpretation == \"MONOCHROME1\":\n        data = np.amax(data) - data\n    data = data - np.min(data)\n    data = data / np.max(data)\n    data = (data * 255).astype(np.uint8)\n    return data","metadata":{"execution":{"iopub.status.busy":"2021-07-04T17:20:44.640405Z","iopub.execute_input":"2021-07-04T17:20:44.640836Z","iopub.status.idle":"2021-07-04T17:20:44.652968Z","shell.execute_reply.started":"2021-07-04T17:20:44.640784Z","shell.execute_reply":"2021-07-04T17:20:44.652107Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Stratified KFold\ndef stratifiedKFold(df,num_folds,random_state):\n    y = df [['Negative for Pneumonia','Typical Appearance', \n            'Indeterminate Appearance', 'Atypical Appearance']]\n    df['fold'] = 0\n    #split data\n    mskf = MultilabelStratifiedKFold(n_splits=num_folds, shuffle= True, random_state=random_state)\n    for i, (_, test_index) in enumerate(mskf.split(df, y)):\n        df.iloc[test_index, -1] = i\n    return df","metadata":{"execution":{"iopub.status.busy":"2021-07-04T17:20:44.654193Z","iopub.execute_input":"2021-07-04T17:20:44.654528Z","iopub.status.idle":"2021-07-04T17:20:44.663318Z","shell.execute_reply.started":"2021-07-04T17:20:44.654491Z","shell.execute_reply":"2021-07-04T17:20:44.662477Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Torch dataset, not trying any augmentations as of now\nclass SIIM_COVID(Dataset):\n    def __init__(self,df,\n                 train = True,\n                 transforms=None,\n                 IMG_SIZE = 256\n                ):\n        self.imageList = df['path'].values\n        self.transform = None\n        if transforms is None:\n            self.transform = A.Compose([\n                                            A.Resize(IMG_SIZE,IMG_SIZE),\n                                            A.Normalize(\n                                                mean=[0.485, 0.456, 0.406],\n                                                std=[0.229, 0.224, 0.225],\n                                            ),\n                                            ToTensorV2()\n                                        ])\n        else:\n            self.transform = transforms\n        self.train  = train    \n        if self.train == True:\n            self.labels = df[LABELS].values\n    \n    def __len__(self):\n        return len(self.imageList)\n    \n    def __getitem__(self,idx):\n        file_path = self.imageList[idx]\n        img = dicom2array(file_path)\n        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n        image = self.transform(image=img)\n        if self.train == True:\n            image = image['image']\n            label = self.labels[idx]\n            return image, label\n        else:\n            return image","metadata":{"execution":{"iopub.status.busy":"2021-07-04T17:20:44.664479Z","iopub.execute_input":"2021-07-04T17:20:44.664841Z","iopub.status.idle":"2021-07-04T17:20:44.679432Z","shell.execute_reply.started":"2021-07-04T17:20:44.664807Z","shell.execute_reply":"2021-07-04T17:20:44.678639Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class EffNet(nn.Module):#Efficientnet defining it seperately, helps in inference where the weights can be loaded seperately instead as a lightning module class \n    def __init__(self,model_name,\n                 num_classes,\n                 pretrained = True\n                ):\n        super().__init__()\n        self.model = timm.create_model(model_name,pretrained = pretrained )\n        n_features = self.model.classifier.in_features\n        self.model.global_pool = nn.Identity()\n        self.model.classifier = nn.Identity()\n        self.pooling = nn.AdaptiveAvgPool2d(1)\n        self.fc = nn.Linear(n_features,num_classes)\n        self.soft = nn.Softmax(dim=1)\n        \n    def forward(self,x): \n        bs = x.size(0) # bs -> batch size\n        features = self.model(x)\n        pooled_features = self.pooling(features).view(bs,-1)\n        output = self.fc(pooled_features) \n        output = self.soft(output)# Keeping a softmax layer to keep the probabilities between 0 and 1 to avoid problems with torch metrics\n\n        return output ","metadata":{"execution":{"iopub.status.busy":"2021-07-04T17:20:44.680585Z","iopub.execute_input":"2021-07-04T17:20:44.681039Z","iopub.status.idle":"2021-07-04T17:20:44.693504Z","shell.execute_reply.started":"2021-07-04T17:20:44.680941Z","shell.execute_reply":"2021-07-04T17:20:44.692682Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class LitSIIM(pl.LightningModule):\n    def __init__(self,model,\n                 df,fold_no,\n                 train_transforms = None,\n                 valid_transforms = None\n                ):\n        super(LitSIIM,self).__init__()\n        self.model = model\n        \n        self.train_transforms = train_transforms \n        self.valid_transforms = valid_transforms\n\n        self.train_dataset = SIIM_COVID(df[df['fold'] != fold_no] , train = True,\n                                        transforms = None, IMG_SIZE = IMG_SIZE)\n        self.valid_dataset = SIIM_COVID(df[df['fold'] == fold_no] , train = True,# To pass both image and targets \n                                        transforms = None, IMG_SIZE = IMG_SIZE)\n                \n        self.train_loss = nn.BCEWithLogitsLoss()\n        self.valid_loss = nn.BCEWithLogitsLoss()\n        \n        self.train_avg_prec = torchmetrics.AveragePrecision(num_classes = 4,pos_label = 1)\n        self.valid_avg_prec = torchmetrics.AveragePrecision(num_classes = 4, pos_label =1)\n        \n        self.learning_rate = 0.001\n        \n    def forward(self,batch):\n        return self.model(batch)\n    \n    def configure_optimizers(self):\n        return torch.optim.Adam(self.model.parameters(), lr=1e-4, weight_decay=3e-6)\n    \n    def prepare_data(self):\n        pass\n    \n    \n    def train_dataloader(self):\n        train_loader = DataLoader(self.train_dataset,\n                                 batch_size =  TRAIN_BATCH_SIZE,\n                                 shuffle = False,\n                                 sampler = None, \n                                 num_workers = os.cpu_count()\n                                 )\n\n        return train_loader\n    \n    def val_dataloader(self):\n        val_loader = DataLoader(self.valid_dataset,\n                               batch_size = VAL_BATCH_SIZE,\n                               shuffle = False,\n                               num_workers = os.cpu_count()\n                               )\n        return val_loader\n    \n    def training_step(self,batch,batch_idx):\n        image,labels = batch \n        logits = self(image)\n        loss = self.train_loss(logits,labels.type_as(logits))\n        train_avg_prec_batch = (2/3) * self.train_avg_prec(logits,labels)\n        self.log(\"train_loss_batch\",loss,prog_bar = True)\n        self.log(\"train_avg_prec_batch\",train_avg_prec_batch,prog_bar = True)\n        return {\n            'loss':loss,\n            'y_pred':logits,\n            'y_true':labels\n        }\n    \n    def training_epoch_end(self,outputs):\n        avg_prec = (2/3) * self.train_avg_prec.compute()\n        print(len(outputs))\n        print(f\"Train Loss Epoch_{self.current_epoch + 1}: {outputs[len(outputs)-1]['loss']}\")\n        print(f\"Train Average Precision Epoch {self.current_epoch +1}:{avg_prec}\")\n        self.log(\"train_avg_prec_epoch\",avg_prec,prog_bar = True)\n    \n    def validation_step(self,batch,batch_idx):\n        image,labels = batch \n        logits = self(image)\n        loss = self.valid_loss(logits,labels.type_as(logits))\n        valid_avg_prec_batch = self.valid_avg_prec(logits,labels)\n        self.log(\"val_loss\",loss,prog_bar = True)\n        self.log(\"val_avg_prec_batch\",(2/3) * valid_avg_prec_batch,prog_bar = True) \n        return {\n            'loss'   : loss,\n            'y_pred' : logits,\n            'target' : labels\n        }\n        \n    def validation_epoch_end(self,outputs):\n        avg_prec = (2/3)*self.valid_avg_prec.compute()\n        print(f\"Train Loss Epoch_{self.current_epoch + 1}: {outputs[len(outputs)-1]['loss']}\")        \n        print(f\"Valid Average Precision Epoch_{self.current_epoch+1} End:{avg_prec}\")\n        self.log(f'val_avg_prec_epoch',avg_prec,prog_bar = True)\n#         PATH = f\"model_state_dict/B5_E{self.current_epoch+1}.pth\"\n#         torch.save(self.model.state_dict(), PATH)\n        return {'val_loss': outputs[0]['loss'], 'val_avg_prec_epoch': avg_prec}\n    \n    def test_step(self,batch,batch_idx):\n        logits = self(batch['image'])\n        return logits\n\n    def test_epoch_end(self, outputs):\n        probs = torch.cat(outputs,dim = 0)\n        probs = probs.detach().cpu().numpy()\n        self.test_predicts = probs  \n        return {'dummy_item': 0}","metadata":{"execution":{"iopub.status.busy":"2021-07-04T17:20:44.698108Z","iopub.execute_input":"2021-07-04T17:20:44.698676Z","iopub.status.idle":"2021-07-04T17:20:44.722843Z","shell.execute_reply.started":"2021-07-04T17:20:44.698633Z","shell.execute_reply":"2021-07-04T17:20:44.722129Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from pytorch_lightning.callbacks.progress import ProgressBar\n\nclass LitProgressBar(ProgressBar):\n    def init_train_tqdm(self):\n        bar = super().init_train_tqdm()\n        bar.leave = True\n        return bar\n        \n    def init_validation_tqdm(self):\n        bar = super().init_validation_tqdm()\n        bar.set_description('Valid')\n        return bar\n\n        \n    def training_epoch_end(self, outputs):\n        self.trainer.progress_bar_callback.main_progress_bar.write(\n            f\"Epoch {self.trainer.current_epoch + 1} training loss={self.trainer.progress_bar_dict['train_loss']}\" +\n            f\"ROC AUC={self.trainer.progress_bar_dict['train_avg_prec_epoch']}\"\n        )\n\n    def validation_epoch_end(self, outputs):\n        loss = torch.stack(outputs).mean()\n        self.trainer.progress_bar_callback.main_progress_bar.write(\n            f\"Epoch {self.trainer.current_epoch + 1} validation loss={self.trainer.progress_bar_dict['val_loss']}\" +\n            f\"Valid ROC AUC= {self.trainer.progress_bar_dict['val_avg_prec_epoch']}\"\n        )\n\n","metadata":{"execution":{"iopub.status.busy":"2021-07-04T17:20:44.724287Z","iopub.execute_input":"2021-07-04T17:20:44.724687Z","iopub.status.idle":"2021-07-04T17:20:44.734538Z","shell.execute_reply.started":"2021-07-04T17:20:44.724654Z","shell.execute_reply":"2021-07-04T17:20:44.733651Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"! mkdir output\n! mkdir model_state_dict # if the model weights are to be stored in model.load_state_dict()  fashion","metadata":{"execution":{"iopub.status.busy":"2021-07-04T17:20:44.735887Z","iopub.execute_input":"2021-07-04T17:20:44.736156Z","iopub.status.idle":"2021-07-04T17:20:46.018578Z","shell.execute_reply.started":"2021-07-04T17:20:44.736133Z","shell.execute_reply":"2021-07-04T17:20:46.017499Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if __name__ == '__main__':\n    df = preprocess_df(train = True)\n    df = stratifiedKFold(df = df,num_folds = NUM_FOLDS,random_state = 42)\n    \n    model = EffNet(\n                    model_name = MODEL_NAME,\n                    num_classes= len(LABELS),\n                    pretrained = True\n                )\n    lit = LitSIIM(\n            model = model,\n            df = df,fold_no = FOLD,\n            train_transforms = None,\n            valid_transforms = None\n    )\n    checkpoint_callback = pl.callbacks.ModelCheckpoint(\n                                        monitor = 'val_avg_prec_epoch',\n                                        dirpath = \"output/\",\n                                        save_top_k = 2, # Change this to the desired number of saves you need. Here the number of epochs is 3 so kept it as 2.\n                                        mode = 'max', # Save models based on the best Validation Average Precision\n                                        filename =  '{epoch}_effnetb5_{val_avg_prec_epoch:.3f}_{val_loss:.3f}',\n    )\n    \n    trainer = pl.Trainer(\n                         max_epochs=EPOCHS, \n                         gradient_clip_val=1,\n                         callbacks=[checkpoint_callback,LitProgressBar()],\n                         gpus=1,\n                         progress_bar_refresh_rate=0\n                     )\n    trainer.fit(lit)","metadata":{"execution":{"iopub.status.busy":"2021-07-04T17:20:46.665087Z","iopub.execute_input":"2021-07-04T17:20:46.665351Z"},"trusted":true},"execution_count":null,"outputs":[]}]}