{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# **UWMGI: Unet PyTorch Training 2.5D Dataset**","metadata":{"papermill":{"duration":0.068881,"end_time":"2022-05-01T14:21:23.855197","exception":false,"start_time":"2022-05-01T14:21:23.786316","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# [UW-Madison GI Tract Image Segmentation](https://www.kaggle.com/competitions/uw-madison-gi-tract-image-segmentation/)\n> Track healthy organs in medical scans to improve cancer treatment\n\n<img src=\"https://storage.googleapis.com/kaggle-competitions/kaggle/27923/logos/header.png?t=2021-06-02-20-30-25\">","metadata":{}},{"cell_type":"markdown","source":"# Methodlogy\n<img src=\"https://i.ibb.co/sgsPf4v/Capture.png\" width=800>\n<img src=\"https://i.ibb.co/KKtZ7Gn/Picture1-3d.png\" width=500>","metadata":{}},{"cell_type":"markdown","source":"* I have used 2.5D image dataset for training with Unet Models with Efficientnet-B0 as backbone of the encoder.\n* 2.5D images take leverage of the extra depth information. \n* Training with 2D images result in bad score/accuracy.\n  3D images will give much better score than 2D and 2.5D images but they are much more computationally heavy and require heavy GPU. Thus 2.5D images are preferred.\n* For segmentation masks data, I have used pre-computed 2.5D masks from this [dataset](https://www.kaggle.com/datasets/awsaf49/uwmgi-mask-dataset)\n* For greater understanding of 2.5D segmentation, you can prefer this link(https://www.arxiv-vanity.com/papers/2010.06163/)\n* For data split, I have used StratifiedGroupFold to avoid data leakage due to `case` and to stratify `empty` and `non-empty` mask cases. ","metadata":{}},{"cell_type":"markdown","source":"# Notebooks\n\n**2.5D**\n* Training : [UWMGI: 2.5D [Train] [PyTorch]](https://www.kaggle.com/awsaf49/uwmgi-2-5d-train-pytorch/)\n* Inference : [UWMGI: 2.5D [Infer] [PyTorch]](https://www.kaggle.com/awsaf49/uwmgi-2-5d-infer-pytorch/)\n* 2.5D : [UWMGI: 2.5D stride=2 Dataset](https://www.kaggle.com/datasets/awsaf49/uwmgi-25d-stride2-dataset)\n\n**UNet**:\n* Train: [UWMGI: Unet [Train] [PyTorch]](https://www.kaggle.com/code/awsaf49/uwmgi-unet-train-pytorch/)\n* Infer: [UWMGI: Unet [Infer] [PyTorch]](https://www.kaggle.com/code/awsaf49/uwmgi-unet-infer-pytorch/)","metadata":{}},{"cell_type":"markdown","source":"# Install Libraries","metadata":{"papermill":{"duration":0.095034,"end_time":"2022-05-01T14:21:24.295469","exception":false,"start_time":"2022-05-01T14:21:24.200435","status":"completed"},"tags":[]}},{"cell_type":"code","source":"!pip install -q segmentation_models_pytorch","metadata":{"execution":{"iopub.status.busy":"2022-07-15T18:07:34.350833Z","iopub.execute_input":"2022-07-15T18:07:34.351296Z","iopub.status.idle":"2022-07-15T18:07:51.768710Z","shell.execute_reply.started":"2022-07-15T18:07:34.351203Z","shell.execute_reply":"2022-07-15T18:07:51.767721Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Import Libraries ","metadata":{"papermill":{"duration":0.051758,"end_time":"2022-05-01T14:21:43.555685","exception":false,"start_time":"2022-05-01T14:21:43.503927","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\npd.options.plotting.backend = \"plotly\"\nimport random\nfrom glob import glob\nimport seaborn as sns\nimport os, shutil\nfrom tqdm import tqdm\ntqdm.pandas()\nimport time\nimport copy\nimport joblib\nfrom collections import defaultdict\nimport gc\nfrom IPython import display as ipd\n\n# visualization\nimport cv2\nimport matplotlib.pyplot as plt\nfrom matplotlib.patches import Rectangle\n\n# Sklearn\nfrom sklearn.model_selection import StratifiedKFold, KFold, StratifiedGroupKFold\n\n# PyTorch \nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.optim as optim\nfrom torch.optim import lr_scheduler\nfrom torch.utils.data import Dataset, DataLoader\nfrom torch.cuda import amp\nimport segmentation_models_pytorch as smp\nimport timm\n\n# Albumentations for augmentations\nimport albumentations as A\nfrom albumentations.pytorch import ToTensorV2\n\nimport rasterio\nfrom joblib import Parallel, delayed\n\n# For colored terminal text\nfrom colorama import Fore, Back, Style\nc_  = Fore.GREEN\nsr_ = Style.RESET_ALL\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")\n\n# For descriptive error messages\nos.environ['CUDA_LAUNCH_BLOCKING'] = \"1\"","metadata":{"_kg_hide-input":false,"papermill":{"duration":11.235988,"end_time":"2022-05-01T14:21:54.843577","exception":false,"start_time":"2022-05-01T14:21:43.607589","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-07-15T18:08:04.612562Z","iopub.execute_input":"2022-07-15T18:08:04.613001Z","iopub.status.idle":"2022-07-15T18:08:19.991006Z","shell.execute_reply.started":"2022-07-15T18:08:04.612964Z","shell.execute_reply":"2022-07-15T18:08:19.989873Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# CFG (Configurations)","metadata":{"papermill":{"duration":0.050231,"end_time":"2022-05-01T14:21:54.946677","exception":false,"start_time":"2022-05-01T14:21:54.896446","status":"completed"},"tags":[]}},{"cell_type":"code","source":"class CFG:\n    seed          = 42\n    debug         = False\n    model_name    = 'Unet'\n    train_bs      = 64  # train batch_size\n    valid_bs      = train_bs*2 # valid batch_size\n    img_size      = [320,384]\n    epochs        = 15\n    lr            = 2e-3 # learning rate\n    scheduler     = 'CosineAnnealingLR' \n    min_lr        = 1e-6\n    T_max         = int(30000/train_bs*epochs)+50\n    T_0           = 25\n    warmup_epochs = 0\n    wd            = 1e-6\n    n_accumulate  = max(1, 32//train_bs)\n    n_fold        = 5\n    fold_selected = 1\n    num_classes   = 3\n    device        = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n    thr           = 0.5 # threshold value\n    folds         = [0]","metadata":{"papermill":{"duration":0.136117,"end_time":"2022-05-01T14:21:55.134348","exception":false,"start_time":"2022-05-01T14:21:54.998231","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-07-15T18:10:34.831943Z","iopub.execute_input":"2022-07-15T18:10:34.832769Z","iopub.status.idle":"2022-07-15T18:10:34.842406Z","shell.execute_reply.started":"2022-07-15T18:10:34.832727Z","shell.execute_reply":"2022-07-15T18:10:34.841072Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"CFG.device","metadata":{"execution":{"iopub.status.busy":"2022-07-15T18:10:41.143632Z","iopub.execute_input":"2022-07-15T18:10:41.144869Z","iopub.status.idle":"2022-07-15T18:10:41.156004Z","shell.execute_reply.started":"2022-07-15T18:10:41.144811Z","shell.execute_reply":"2022-07-15T18:10:41.154764Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Reproducibilty","metadata":{}},{"cell_type":"code","source":"def set_seed(seed = 42):\n    '''Sets the seed of the entire notebook so results are the same every time we run.\n    This is for REPRODUCIBILITY.'''\n    np.random.seed(seed)\n    random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    # When running on the CuDNN backend, two further options must be set\n    torch.backends.cudnn.deterministic = True\n    torch.backends.cudnn.benchmark = False\n    # Set a fixed value for the hash seed\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    print('> SEEDING DONE')\n    \nset_seed(CFG.seed)","metadata":{"papermill":{"duration":0.067447,"end_time":"2022-05-01T14:21:55.253013","exception":false,"start_time":"2022-05-01T14:21:55.185566","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-07-15T18:11:21.679197Z","iopub.execute_input":"2022-07-15T18:11:21.679594Z","iopub.status.idle":"2022-07-15T18:11:21.690092Z","shell.execute_reply.started":"2022-07-15T18:11:21.679565Z","shell.execute_reply":"2022-07-15T18:11:21.689067Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Meta Data \n* Files\n    * train.csv - IDs and masks for all training objects.\n    * sample_submission.csv - a sample submission file in the correct format\n    * train - a folder of case/day folders, each containing slice images for a particular case on a given day.\n\n> Note that the image filenames include 4 numbers (ex. `276_276_1.63_1.63.png`). These four numbers are slice height / width (integers in pixels) and heigh/width pixel spacing (floating points in mm). The first two defines the resolution of the slide. The last two record the physical size of each pixel.\n\n* Columns\n    * id - unique identifier for object\n    * class - the predicted class for the object\n    * EncodedPixels - RLE-encoded pixels for the identified object","metadata":{}},{"cell_type":"code","source":"path_df = pd.DataFrame(glob('/kaggle/input/uwmgi-25d-stride2-dataset/images/images/*'), columns=['image_path']) # Writing paths of all the 2.5D images in the path_df dataframe and renaming column as 'image_path'\npath_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-15T18:14:54.071904Z","iopub.execute_input":"2022-07-15T18:14:54.072712Z","iopub.status.idle":"2022-07-15T18:14:54.250954Z","shell.execute_reply.started":"2022-07-15T18:14:54.072642Z","shell.execute_reply":"2022-07-15T18:14:54.249679Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"path_df.image_path.iloc[0]","metadata":{"execution":{"iopub.status.busy":"2022-07-15T18:16:05.381394Z","iopub.execute_input":"2022-07-15T18:16:05.381773Z","iopub.status.idle":"2022-07-15T18:16:05.389134Z","shell.execute_reply.started":"2022-07-15T18:16:05.381740Z","shell.execute_reply":"2022-07-15T18:16:05.387974Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"path_df['mask_path'] = path_df.image_path.str.replace('image','mask') # replacing all the words of image_path containing 'image' as string with 'mask' and renaming as 'mask_path'\npath_df['id'] = path_df.image_path.map(lambda x: x.split('/')[-1].replace('.npy','')) # splitting the image_paths and storing the last index of this split containing information of an image like  case130_day20_slice_0131.\npath_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-15T18:16:40.003518Z","iopub.execute_input":"2022-07-15T18:16:40.003917Z","iopub.status.idle":"2022-07-15T18:16:40.109557Z","shell.execute_reply.started":"2022-07-15T18:16:40.003885Z","shell.execute_reply":"2022-07-15T18:16:40.108500Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = pd.read_csv('../input/uwmgi-mask-dataset/train.csv')\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-15T18:20:14.487139Z","iopub.execute_input":"2022-07-15T18:20:14.487508Z","iopub.status.idle":"2022-07-15T18:20:15.771532Z","shell.execute_reply.started":"2022-07-15T18:20:14.487478Z","shell.execute_reply":"2022-07-15T18:20:15.770353Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['segmentation'] = df.segmentation.fillna('') # filling NaN values with ''\ndf['rle_len'] = df.segmentation.map(len) # length of each rle mask\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-15T18:21:02.280981Z","iopub.execute_input":"2022-07-15T18:21:02.281816Z","iopub.status.idle":"2022-07-15T18:21:02.382041Z","shell.execute_reply.started":"2022-07-15T18:21:02.281757Z","shell.execute_reply":"2022-07-15T18:21:02.380690Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df2 = df.groupby(['id'])['segmentation'].agg(list).to_frame().reset_index() # grouping the df dataframe by id such that all rle values of each id can be accesses and then converted them to lists\ndf2.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-15T18:23:40.266417Z","iopub.execute_input":"2022-07-15T18:23:40.266830Z","iopub.status.idle":"2022-07-15T18:23:40.691839Z","shell.execute_reply.started":"2022-07-15T18:23:40.266795Z","shell.execute_reply":"2022-07-15T18:23:40.690821Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df2 = df2.merge(df.groupby(['id'])['rle_len'].agg(sum).to_frame().reset_index()) # total length of all rles of each id\ndf2.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-15T18:23:57.826919Z","iopub.execute_input":"2022-07-15T18:23:57.827317Z","iopub.status.idle":"2022-07-15T18:23:57.967350Z","shell.execute_reply.started":"2022-07-15T18:23:57.827284Z","shell.execute_reply":"2022-07-15T18:23:57.965986Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = df.drop(columns=['segmentation', 'class', 'rle_len'])\ndf = df.groupby(['id']).head(1).reset_index(drop=True)\ndf = df.merge(df2, on=['id'])\ndf['empty'] = (df.rle_len==0) # empty masks\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-15T18:24:45.363441Z","iopub.execute_input":"2022-07-15T18:24:45.364186Z","iopub.status.idle":"2022-07-15T18:24:45.522585Z","shell.execute_reply.started":"2022-07-15T18:24:45.364134Z","shell.execute_reply":"2022-07-15T18:24:45.521193Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = df.drop(columns=['image_path','mask_path'])\ndf = df.merge(path_df, on=['id'])\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-15T18:24:53.470464Z","iopub.execute_input":"2022-07-15T18:24:53.470854Z","iopub.status.idle":"2022-07-15T18:24:53.546870Z","shell.execute_reply.started":"2022-07-15T18:24:53.470822Z","shell.execute_reply":"2022-07-15T18:24:53.545682Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Removing Faulty Datas\n#### For more information , refer to this [discussion](https://www.kaggle.com/competitions/uw-madison-gi-tract-image-segmentation/discussion/321979)","metadata":{}},{"cell_type":"code","source":"fault1 = 'case7_day0'\nfault2 = 'case81_day30'\ndf = df[~df['id'].str.contains(fault1) & ~df['id'].str.contains(fault2)].reset_index(drop=True)\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-15T18:32:38.591658Z","iopub.execute_input":"2022-07-15T18:32:38.592579Z","iopub.status.idle":"2022-07-15T18:32:38.712811Z","shell.execute_reply.started":"2022-07-15T18:32:38.592520Z","shell.execute_reply":"2022-07-15T18:32:38.711747Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Class Distribution","metadata":{}},{"cell_type":"code","source":"df_train = df.copy()","metadata":{"execution":{"iopub.status.busy":"2022-07-15T18:33:08.567500Z","iopub.execute_input":"2022-07-15T18:33:08.567920Z","iopub.status.idle":"2022-07-15T18:33:08.576112Z","shell.execute_reply.started":"2022-07-15T18:33:08.567884Z","shell.execute_reply":"2022-07-15T18:33:08.575224Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train['empty'].value_counts().plot.bar()","metadata":{"execution":{"iopub.status.busy":"2022-07-15T18:33:09.263560Z","iopub.execute_input":"2022-07-15T18:33:09.264184Z","iopub.status.idle":"2022-07-15T18:33:11.487366Z","shell.execute_reply.started":"2022-07-15T18:33:09.264149Z","shell.execute_reply":"2022-07-15T18:33:11.486187Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(12,4))\nbar = plt.bar([1,2,3],100*np.mean( df_train.iloc[:,1:4]!='',axis=0))\nplt.title('Percent Training Images with Mask', fontsize=16)\nplt.ylabel('Percent of Images'); plt.xlabel('Class Type')\nplt.xticks([1,2,3])\nlabels=[\"large bowel\",\"small bowel\",\"stomach\"]\nfor rect,lbl in zip(bar,labels):\n    height = rect.get_height()\n    plt.text(rect.get_x() + rect.get_width()/3, height,  lbl,\n             ha='center', va='bottom',fontsize=16)\n    plt.text(rect.get_x() + rect.get_width()/1.3, height, '%.1f %%' % height,\n             ha='center', va='bottom',fontsize=13)\n\nplt.ylim((0,50)); plt.show()","metadata":{"papermill":{"duration":0.308924,"end_time":"2022-05-01T14:22:04.697458","exception":false,"start_time":"2022-05-01T14:22:04.388534","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-07-15T18:33:15.942180Z","iopub.execute_input":"2022-07-15T18:33:15.942582Z","iopub.status.idle":"2022-07-15T18:33:16.198977Z","shell.execute_reply.started":"2022-07-15T18:33:15.942551Z","shell.execute_reply":"2022-07-15T18:33:16.197726Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Generating masks from Encoded Pixels\n#### For understanding about Encoded Pixels , refer this [medium](https://medium.com/analytics-vidhya/generating-masks-from-encoded-pixels-semantic-segmentation-18635e834ad0) article on encoded pixels.","metadata":{"papermill":{"duration":0.057618,"end_time":"2022-05-01T14:22:04.811513","exception":false,"start_time":"2022-05-01T14:22:04.753895","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# ref: https://www.kaggle.com/paulorzp/run-length-encode-and-decode\ndef rle_decode(mask_rle, shape):\n    '''\n    mask_rle: run-length as string formated (start length)\n    shape: (height,width) of array to return \n    Returns numpy array, 1 - mask, 0 - background\n\n    '''\n    s = mask_rle.split() # splitting the rle lists\n    starts, lengths = [np.asarray(x, dtype=int) for x in (s[0:][::2], s[1:][::2])] # making two lists starts and lengths , starts contains\n    # all the starting pixels and lengths contains information on taking how many pixels starting from the pixel in starts list at the same index\n    # For ex : starts = [231,564] , lengths = [5,8] -> for index = 0 -> take 5 pixels starting from 231\n    starts -= 1 \n    ends = starts + lengths # contains the ending pixels\n    img = np.zeros(shape[0]*shape[1], dtype=np.uint8)\n    for lo, hi in zip(starts, ends):\n        img[lo:hi] = 1 # marking all the rle encoded pixels in img numpy array as 1\n    return img.reshape(shape)  # Needed to align to RLE direction (reshaping)\n\n\n# ref.: https://www.kaggle.com/stainsby/fast-tested-rle\ndef rle_encode(img):\n    '''\n    img: numpy array, 1 - mask, 0 - background\n    Returns run length as string formated\n    '''\n    pixels = img.flatten()\n    pixels = np.concatenate([[0], pixels, [0]])\n    runs = np.where(pixels[1:] != pixels[:-1])[0] + 1\n    runs[1::2] -= runs[::2]\n    return ' '.join(str(x) for x in runs)\n\n\ndef show_img(img, mask=None):\n    clahe = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8,8))\n    plt.imshow(img, cmap='bone')\n    \n    if mask is not None:\n        plt.imshow(mask, alpha=0.5)\n        handles = [Rectangle((0,0),1,1, color=_c) for _c in [(0.667,0.0,0.0), (0.0,0.667,0.0), (0.0,0.0,0.667)]]\n        labels = [\"Large Bowel\", \"Small Bowel\", \"Stomach\"]\n        plt.legend(handles,labels)\n    plt.axis('off')","metadata":{"papermill":{"duration":0.071865,"end_time":"2022-05-01T14:22:04.941636","exception":false,"start_time":"2022-05-01T14:22:04.869771","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-07-15T18:42:21.234345Z","iopub.execute_input":"2022-07-15T18:42:21.234836Z","iopub.status.idle":"2022-07-15T18:42:21.251156Z","shell.execute_reply.started":"2022-07-15T18:42:21.234802Z","shell.execute_reply":"2022-07-15T18:42:21.249808Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Create Folds","metadata":{"papermill":{"duration":0.053777,"end_time":"2022-05-01T14:22:05.050418","exception":false,"start_time":"2022-05-01T14:22:04.996641","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# Splitting the data into 5 folds (CFG.n_fold) using StratifiedGroupKFold\nskf = StratifiedGroupKFold(n_splits=CFG.n_fold, shuffle=True, random_state=CFG.seed)\nfor fold, (train_idx, val_idx) in enumerate(skf.split(df_train, df_train['empty'], groups = df_train[\"case\"])):\n    df_train.loc[val_idx, 'fold'] = fold\ndf_train.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-15T18:44:01.405242Z","iopub.execute_input":"2022-07-15T18:44:01.406053Z","iopub.status.idle":"2022-07-15T18:44:01.600564Z","shell.execute_reply.started":"2022-07-15T18:44:01.406013Z","shell.execute_reply":"2022-07-15T18:44:01.599200Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_train.groupby(['fold','empty'])['id'].count())","metadata":{"execution":{"iopub.status.busy":"2022-07-15T18:44:06.671247Z","iopub.execute_input":"2022-07-15T18:44:06.671692Z","iopub.status.idle":"2022-07-15T18:44:06.699772Z","shell.execute_reply.started":"2022-07-15T18:44:06.671632Z","shell.execute_reply":"2022-07-15T18:44:06.698534Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Dataset","metadata":{"papermill":{"duration":0.056645,"end_time":"2022-05-01T14:22:05.689182","exception":false,"start_time":"2022-05-01T14:22:05.632537","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def load_img(path):\n#         print(\"path \",path)\n        img = np.load(path)\n        img = img.astype('float32') # original is uint16\n        mx = np.max(img)\n        if mx:\n            img/=mx # scale image to [0, 1]\n        return img\n\ndef load_msk(path):\n    msk = np.load(path)\n    msk = msk.astype('float32')\n    msk/=255.0\n    return msk","metadata":{"execution":{"iopub.status.busy":"2022-07-15T18:44:39.091292Z","iopub.execute_input":"2022-07-15T18:44:39.091727Z","iopub.status.idle":"2022-07-15T18:44:39.098749Z","shell.execute_reply.started":"2022-07-15T18:44:39.091688Z","shell.execute_reply":"2022-07-15T18:44:39.097782Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class BuildDataset(torch.utils.data.Dataset):\n    def __init__(self, df, label=True, transforms=None):\n        self.df         = df\n        self.label      = label\n        self.img_paths  = df['image_path'].tolist()\n        self.msk_paths  = df['mask_path'].tolist()\n        self.transforms = transforms\n        \n    def __len__(self):\n        return len(self.df)\n    \n    def __getitem__(self, index): # iterating to all rows of df\n        img_path  = self.img_paths[index] \n        img = []\n        img = load_img(img_path)\n        \n        if self.label:\n            msk_path = self.msk_paths[index]\n            msk = load_msk(msk_path)\n            if self.transforms:\n                data = self.transforms(image=img, mask=msk)\n                img  = data['image']\n                msk  = data['mask']\n            img = np.transpose(img, (2, 0, 1))\n            msk = np.transpose(msk, (2, 0, 1))\n            return torch.tensor(img), torch.tensor(msk)\n        else:\n            if self.transforms:\n                data = self.transforms(image=img)\n                img  = data['image']\n            img = np.transpose(img, (2, 0, 1))\n            return torch.tensor(img)","metadata":{"execution":{"iopub.status.busy":"2022-07-15T18:46:07.947061Z","iopub.execute_input":"2022-07-15T18:46:07.948191Z","iopub.status.idle":"2022-07-15T18:46:07.960540Z","shell.execute_reply.started":"2022-07-15T18:46:07.948144Z","shell.execute_reply":"2022-07-15T18:46:07.959633Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Augmentations","metadata":{"papermill":{"duration":0.058218,"end_time":"2022-05-01T14:22:05.941496","exception":false,"start_time":"2022-05-01T14:22:05.883278","status":"completed"},"tags":[]}},{"cell_type":"code","source":"data_transforms = {\n    \"train\": A.Compose([\n#         A.Resize(*CFG.img_size, interpolation=cv2.INTER_NEAREST),\n        A.HorizontalFlip(p=0.5),\n#         A.VerticalFlip(p=0.5),\n        A.ShiftScaleRotate(shift_limit=0.0625, scale_limit=0.05, rotate_limit=10, p=0.5),\n        A.OneOf([\n            A.GridDistortion(num_steps=5, distort_limit=0.05, p=1.0),\n# #             A.OpticalDistortion(distort_limit=0.05, shift_limit=0.05, p=1.0),\n            A.ElasticTransform(alpha=1, sigma=50, alpha_affine=50, p=1.0)\n        ], p=0.25),\n        A.CoarseDropout(max_holes=8, max_height=CFG.img_size[0]//20, max_width=CFG.img_size[1]//20,\n                         min_holes=5, fill_value=0, mask_fill_value=0, p=0.5),\n        ], p=1.0),\n    \n    \"valid\": A.Compose([\n#         A.Resize(*CFG.img_size, interpolation=cv2.INTER_NEAREST),\n        ], p=1.0)\n}","metadata":{"execution":{"iopub.status.busy":"2022-07-15T18:47:14.853262Z","iopub.execute_input":"2022-07-15T18:47:14.853707Z","iopub.status.idle":"2022-07-15T18:47:14.863239Z","shell.execute_reply.started":"2022-07-15T18:47:14.853654Z","shell.execute_reply":"2022-07-15T18:47:14.862264Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# DataLoader","metadata":{"papermill":{"duration":0.056669,"end_time":"2022-05-01T14:22:06.179941","exception":false,"start_time":"2022-05-01T14:22:06.123272","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def prepare_loaders(fold, debug=False):\n    train_df = df_train.query(\"fold!=@fold\").reset_index(drop=True)\n    valid_df = df_train.query(\"fold==@fold\").reset_index(drop=True)\n    if debug:\n        train_df = train_df.head(32*5).query(\"empty==0\")\n        valid_df = valid_df.head(32*3).query(\"empty==0\")\n    train_dataset = BuildDataset(train_df, transforms=data_transforms['train'])\n    valid_dataset = BuildDataset(valid_df, transforms=data_transforms['valid'])\n\n    train_loader = DataLoader(train_dataset, batch_size=CFG.train_bs if not debug else 20, \n                              num_workers=4, shuffle=True, pin_memory=True, drop_last=False) \n    valid_loader = DataLoader(valid_dataset, batch_size=CFG.valid_bs if not debug else 20, \n                              num_workers=4, shuffle=False, pin_memory=True)\n    \n    return train_loader, valid_loader","metadata":{"execution":{"iopub.status.busy":"2022-07-15T18:47:22.105127Z","iopub.execute_input":"2022-07-15T18:47:22.105741Z","iopub.status.idle":"2022-07-15T18:47:22.114120Z","shell.execute_reply.started":"2022-07-15T18:47:22.105709Z","shell.execute_reply":"2022-07-15T18:47:22.112943Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_loader, valid_loader = prepare_loaders(fold=0, debug=True)","metadata":{"execution":{"iopub.status.busy":"2022-07-15T18:47:39.357236Z","iopub.execute_input":"2022-07-15T18:47:39.357848Z","iopub.status.idle":"2022-07-15T18:47:39.409221Z","shell.execute_reply.started":"2022-07-15T18:47:39.357814Z","shell.execute_reply":"2022-07-15T18:47:39.408250Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"imgs, msks = next(iter(train_loader))\nimgs.size(), msks.size()","metadata":{"execution":{"iopub.status.busy":"2022-07-15T18:47:41.945506Z","iopub.execute_input":"2022-07-15T18:47:41.946321Z","iopub.status.idle":"2022-07-15T18:47:44.272753Z","shell.execute_reply.started":"2022-07-15T18:47:41.946271Z","shell.execute_reply":"2022-07-15T18:47:44.271148Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#  Visualization","metadata":{"papermill":{"duration":0.078956,"end_time":"2022-05-01T14:22:14.076464","exception":false,"start_time":"2022-05-01T14:22:13.997508","status":"completed"},"tags":[]}},{"cell_type":"code","source":"imgs, msks = next(iter(train_loader))\nimgs.size(), msks.size()\n\ndef plot_batch(imgs, msks, size=3):\n    plt.figure(figsize=(5*5, 5))\n    for idx in range(size):\n        plt.subplot(1, 5, idx+1)\n        img = imgs[idx,].permute((1, 2, 0)).numpy()\n        msk = msks[idx,].permute((1, 2, 0)).numpy()\n        show_img(img, msk)\n    plt.tight_layout()\n    plt.show()\n\nplot_batch(imgs, msks, size=5)","metadata":{"_kg_hide-input":true,"papermill":{"duration":3.628283,"end_time":"2022-05-01T14:22:17.817956","exception":false,"start_time":"2022-05-01T14:22:14.189673","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-07-15T18:48:07.148088Z","iopub.execute_input":"2022-07-15T18:48:07.148579Z","iopub.status.idle":"2022-07-15T18:48:11.140735Z","shell.execute_reply.started":"2022-07-15T18:48:07.148539Z","shell.execute_reply":"2022-07-15T18:48:11.139769Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gc.collect()","metadata":{"papermill":{"duration":0.343671,"end_time":"2022-05-01T14:22:18.227067","exception":false,"start_time":"2022-05-01T14:22:17.883396","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-07-15T18:48:11.552615Z","iopub.execute_input":"2022-07-15T18:48:11.553049Z","iopub.status.idle":"2022-07-15T18:48:12.115514Z","shell.execute_reply.started":"2022-07-15T18:48:11.553017Z","shell.execute_reply":"2022-07-15T18:48:12.114211Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# UNet Model\n","metadata":{"papermill":{"duration":0.065659,"end_time":"2022-05-01T14:22:18.364383","exception":false,"start_time":"2022-05-01T14:22:18.298724","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"#### For using different encoders and decoders , prefer to this github repository [segmentation_model_pytorch](https://github.com/qubvel/segmentation_models.pytorch) ","metadata":{}},{"cell_type":"code","source":"def build_model():\n    model = smp.Unet(\"efficientnet-b0\", encoder_weights=\"imagenet\",in_channels=3, classes=3,activation=None) # Transfer learning with image_nets weights\n    model.to(CFG.device)\n    return model\n\n\ndef load_model(path):\n    model = build_model()\n    model.load_state_dict(torch.load(path))\n    model.eval()\n    return model","metadata":{"papermill":{"duration":0.075448,"end_time":"2022-05-01T14:22:18.660525","exception":false,"start_time":"2022-05-01T14:22:18.585077","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-07-15T18:50:15.533384Z","iopub.execute_input":"2022-07-15T18:50:15.534316Z","iopub.status.idle":"2022-07-15T18:50:15.543026Z","shell.execute_reply.started":"2022-07-15T18:50:15.534264Z","shell.execute_reply":"2022-07-15T18:50:15.541654Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# DeepLab V3","metadata":{}},{"cell_type":"code","source":"# def build_model():\n#     model = smp.DeepLabV3(encoder_name=\"resnet50\",encoder_weights='imagenet',classes = 3,activation=None)\n#     model.to(CFG.device)\n#     return model\n\n\n# def load_model(path):\n#     model = build_model()\n#     model.load_state_dict(torch.load(path))\n#     model.eval()\n#     return model","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:37:59.356669Z","iopub.execute_input":"2022-07-12T10:37:59.357319Z","iopub.status.idle":"2022-07-12T10:37:59.369657Z","shell.execute_reply.started":"2022-07-12T10:37:59.357269Z","shell.execute_reply":"2022-07-12T10:37:59.368376Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 🔧 Loss Function","metadata":{"papermill":{"duration":0.065476,"end_time":"2022-05-01T14:22:18.791235","exception":false,"start_time":"2022-05-01T14:22:18.725759","status":"completed"},"tags":[]}},{"cell_type":"code","source":"JaccardLoss = smp.losses.JaccardLoss(mode='multilabel')\nDiceLoss    = smp.losses.DiceLoss(mode='multilabel')\nBCELoss     = smp.losses.SoftBCEWithLogitsLoss()\nLovaszLoss  = smp.losses.LovaszLoss(mode='multilabel', per_image=False)\nTverskyLoss = smp.losses.TverskyLoss(mode='multilabel', log_loss=False)\nFocalloss = smp.losses.FocalLoss(mode='multilabel')\n\ndef dice_coef(y_true, y_pred, thr=0.5, dim=(2,3), epsilon=0.001):\n    y_true = y_true.to(torch.float32)\n    y_pred = (y_pred>thr).to(torch.float32)\n    inter = (y_true*y_pred).sum(dim=dim) \n    den = y_true.sum(dim=dim) + y_pred.sum(dim=dim)\n    dice = ((2*inter+epsilon)/(den+epsilon)).mean(dim=(1,0))\n    return dice\n\ndef iou_coef(y_true, y_pred, thr=0.5, dim=(2,3), epsilon=0.001):\n    y_true = y_true.to(torch.float32)\n    y_pred = (y_pred>thr).to(torch.float32)\n    inter = (y_true*y_pred).sum(dim=dim)\n    union = (y_true + y_pred - y_true*y_pred).sum(dim=dim)\n    iou = ((inter+epsilon)/(union+epsilon)).mean(dim=(1,0))\n    return iou\n\ndef criterion(y_pred, y_true):\n    return  0.5*BCELoss(y_pred,y_true) + 0.5*TverskyLoss(y_pred, y_true)\n# 0.5*BCELoss(y_pred, y_true) + 0.5*TverskyLoss(y_pred, y_true)\n# 0.6*BCELoss(y_pred, y_true) + 0.4*DiceLoss(y_pred, y_true)+\n# Focalloss(y_pred,y_true)","metadata":{"papermill":{"duration":0.082752,"end_time":"2022-05-01T14:22:18.940904","exception":false,"start_time":"2022-05-01T14:22:18.858152","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-07-15T18:50:27.550816Z","iopub.execute_input":"2022-07-15T18:50:27.552085Z","iopub.status.idle":"2022-07-15T18:50:27.566028Z","shell.execute_reply.started":"2022-07-15T18:50:27.551995Z","shell.execute_reply":"2022-07-15T18:50:27.564866Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Training Function","metadata":{"papermill":{"duration":0.099375,"end_time":"2022-05-01T14:22:19.108356","exception":false,"start_time":"2022-05-01T14:22:19.008981","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def train_one_epoch(model, optimizer, scheduler, dataloader, device, epoch):\n    model.train()\n    scaler = amp.GradScaler()\n    \n    dataset_size = 0\n    running_loss = 0.0\n    train_scores = []\n    pbar = tqdm(enumerate(dataloader), total=len(dataloader), desc='Train ') \n    for step, (images, masks) in pbar:         \n        images = images.to(device, dtype=torch.float)\n        masks  = masks.to(device, dtype=torch.float)\n        \n        batch_size = images.size(0)\n        \n        with amp.autocast(enabled=True):\n            y_pred = model(images)\n            loss   = criterion(y_pred, masks)\n            loss   = loss / CFG.n_accumulate\n            \n        scaler.scale(loss).backward()\n        y_pred_temp = nn.Sigmoid()(y_pred)\n        train_dice = dice_coef(masks, y_pred_temp).cpu().detach().numpy()\n        train_jaccard = iou_coef(masks, y_pred_temp).cpu().detach().numpy()\n        train_scores.append([train_dice, train_jaccard])\n    \n        if (step + 1) % CFG.n_accumulate == 0:\n            scaler.step(optimizer)\n            scaler.update()\n\n                # zero the parameter gradients\n            optimizer.zero_grad()\n\n            if scheduler is not None:\n                scheduler.step()\n                \n        running_loss += (loss.item() * batch_size)\n        dataset_size += batch_size\n        \n        epoch_loss = running_loss / dataset_size\n        \n        mem = torch.cuda.memory_reserved() / 1E9 if torch.cuda.is_available() else 0\n        current_lr = optimizer.param_groups[0]['lr']\n        pbar.set_postfix(train_loss=f'{epoch_loss:0.4f}',\n                        lr=f'{current_lr:0.5f}',\n                        gpu_mem=f'{mem:0.2f} GB')\n    train_scores  = np.mean(train_scores, axis=0)\n    torch.cuda.empty_cache()\n    gc.collect()\n    \n    return epoch_loss , train_scores","metadata":{"papermill":{"duration":0.137479,"end_time":"2022-05-01T14:22:19.392561","exception":false,"start_time":"2022-05-01T14:22:19.255082","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-07-15T18:50:58.167452Z","iopub.execute_input":"2022-07-15T18:50:58.167864Z","iopub.status.idle":"2022-07-15T18:50:58.183618Z","shell.execute_reply.started":"2022-07-15T18:50:58.167831Z","shell.execute_reply":"2022-07-15T18:50:58.182626Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Validation Function","metadata":{"papermill":{"duration":0.110867,"end_time":"2022-05-01T14:22:19.616474","exception":false,"start_time":"2022-05-01T14:22:19.505607","status":"completed"},"tags":[]}},{"cell_type":"code","source":"@torch.no_grad()\ndef valid_one_epoch(model, dataloader, device, epoch):\n    model.eval()\n    \n    dataset_size = 0\n    running_loss = 0.0\n    \n    val_scores = []\n    \n    pbar = tqdm(enumerate(dataloader), total=len(dataloader), desc='Valid ')\n\n    for step, (images, masks) in pbar:        \n        images  = images.to(device, dtype=torch.float)\n        masks   = masks.to(device, dtype=torch.float)\n        \n        batch_size = images.size(0)\n        \n        y_pred  = model(images)\n        loss    = criterion(y_pred, masks)\n        \n        running_loss += (loss.item() * batch_size)\n        dataset_size += batch_size\n        \n        epoch_loss = running_loss / dataset_size\n        \n        y_pred = nn.Sigmoid()(y_pred)\n        val_dice = dice_coef(masks, y_pred).cpu().detach().numpy()\n        val_jaccard = iou_coef(masks, y_pred).cpu().detach().numpy()\n        val_scores.append([val_dice, val_jaccard])\n        \n        mem = torch.cuda.memory_reserved() / 1E9 if torch.cuda.is_available() else 0\n        current_lr = optimizer.param_groups[0]['lr']\n        pbar.set_postfix(valid_loss=f'{epoch_loss:0.4f}',\n                        lr=f'{current_lr:0.5f}',\n                        gpu_memory=f'{mem:0.2f} GB')\n    val_scores  = np.mean(val_scores, axis=0)\n    torch.cuda.empty_cache()\n    gc.collect()\n    \n    return epoch_loss, val_scores","metadata":{"papermill":{"duration":0.081551,"end_time":"2022-05-01T14:22:19.808831","exception":false,"start_time":"2022-05-01T14:22:19.72728","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-07-15T18:51:43.534839Z","iopub.execute_input":"2022-07-15T18:51:43.535418Z","iopub.status.idle":"2022-07-15T18:51:43.553884Z","shell.execute_reply.started":"2022-07-15T18:51:43.535367Z","shell.execute_reply":"2022-07-15T18:51:43.552443Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training","metadata":{"papermill":{"duration":0.065922,"end_time":"2022-05-01T14:22:19.942521","exception":false,"start_time":"2022-05-01T14:22:19.876599","status":"completed"},"tags":[],"_kg_hide-input":true}},{"cell_type":"code","source":"def run_training(model, optimizer, scheduler, device, num_epochs):\n    # To automatically log gradients\n\n    \n    if torch.cuda.is_available():\n        print(\"cuda: {}\\n\".format(torch.cuda.get_device_name()))\n    \n    start = time.time()\n    best_model_wts = copy.deepcopy(model.state_dict()) # so that original model weights dont get changed and we can use orginal model later as well\n    best_dice      = -np.inf\n    best_epoch     = -1\n    history = defaultdict(list)\n    for epoch in range(1, num_epochs + 1): \n        gc.collect()\n        print(f'Epoch {epoch}/{num_epochs}', end='')\n        train_loss,train_scoring = train_one_epoch(model, optimizer, scheduler, \n                                           dataloader=train_loader, \n                                           device=CFG.device, epoch=epoch)\n        \n        train_dice , train_jaccard = train_scoring\n        \n        val_loss, val_scores = valid_one_epoch(model, valid_loader, \n                                                 device=CFG.device, \n                                                 epoch=epoch)\n        \n        val_dice, val_jaccard = val_scores\n        \n    \n        history['Train Loss'].append(train_loss)\n        history['Train Dice'].append(train_dice)\n        history['Train Jaccard'].append(train_jaccard)\n        history['Valid Loss'].append(val_loss)\n        history['Valid Dice'].append(val_dice)\n        history['Valid Jaccard'].append(val_jaccard)\n        \n        # Log the metrics\n\n        \n        print(f'Valid Dice: {val_dice:0.4f} | Valid Jaccard: {val_jaccard:0.4f} | Train Dice: {train_dice:0.4f} | Train Jaccard: {train_jaccard:0.4f}')\n        \n        # deep copy the model\n        if val_dice >= best_dice:\n            print(f\"{c_}Valid Score Improved ({best_dice:0.4f} ---> {val_dice:0.4f})\")\n            best_dice    = val_dice\n            best_jaccard = val_jaccard\n            best_epoch   = epoch\n            #run.summary[\"Best Dice\"]    = best_dice\n           # run.summary[\"Best Jaccard\"] = best_jaccard\n           # run.summary[\"Best Epoch\"]   = best_epoch\n            best_model_wts = copy.deepcopy(model.state_dict())\n            PATH = f\"best_epoch-{fold:02d}.bin\"\n            torch.save(model.state_dict(), PATH)\n            # Save a model file from the current directory\n            print(f\"Model Saved{sr_}\")\n            \n        last_model_wts = copy.deepcopy(model.state_dict())\n        PATH = f\"last_epoch-{fold:02d}.bin\"\n        torch.save(model.state_dict(), PATH)\n            \n        print(); print()\n    \n    end = time.time()\n    time_elapsed = end - start\n    print('Training complete in {:.0f}h {:.0f}m {:.0f}s'.format(\n        time_elapsed // 3600, (time_elapsed % 3600) // 60, (time_elapsed % 3600) % 60))\n    print(\"Best Score: {:.4f}\".format(best_jaccard))\n    \n    # load best model weights\n    model.load_state_dict(best_model_wts)\n    \n    return model, history","metadata":{"papermill":{"duration":0.081613,"end_time":"2022-05-01T14:22:20.088511","exception":false,"start_time":"2022-05-01T14:22:20.006898","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-07-15T18:52:36.135604Z","iopub.execute_input":"2022-07-15T18:52:36.136097Z","iopub.status.idle":"2022-07-15T18:52:36.155133Z","shell.execute_reply.started":"2022-07-15T18:52:36.136057Z","shell.execute_reply":"2022-07-15T18:52:36.153768Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def fetch_scheduler(optimizer):\n    if CFG.scheduler == 'CosineAnnealingLR':\n        scheduler = lr_scheduler.CosineAnnealingLR(optimizer,T_max=CFG.T_max, \n                                                   eta_min=CFG.min_lr)\n    elif CFG.scheduler == 'CosineAnnealingWarmRestarts':\n        scheduler = lr_scheduler.CosineAnnealingWarmRestarts(optimizer,T_0=CFG.T_0, \n                                                             eta_min=CFG.min_lr)\n    elif CFG.scheduler == 'ReduceLROnPlateau':\n        scheduler = lr_scheduler.ReduceLROnPlateau(optimizer,\n                                                   mode='min',\n                                                   factor=0.1,\n                                                   patience=7,\n                                                   threshold=0.0001,\n                                                   min_lr=CFG.min_lr,)\n    elif CFG.scheduer == 'ExponentialLR':\n        scheduler = lr_scheduler.ExponentialLR(optimizer, gamma=0.85)\n    elif CFG.scheduler == None:\n        return None\n        \n    return scheduler","metadata":{"papermill":{"duration":0.075722,"end_time":"2022-05-01T14:22:20.229967","exception":false,"start_time":"2022-05-01T14:22:20.154245","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-07-15T18:52:38.612916Z","iopub.execute_input":"2022-07-15T18:52:38.613965Z","iopub.status.idle":"2022-07-15T18:52:38.622898Z","shell.execute_reply.started":"2022-07-15T18:52:38.613925Z","shell.execute_reply":"2022-07-15T18:52:38.621739Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = build_model()\noptimizer = optim.AdamW(model.parameters(), lr=CFG.lr, weight_decay=CFG.wd)\nscheduler = fetch_scheduler(optimizer)","metadata":{"papermill":{"duration":0.229539,"end_time":"2022-05-01T14:22:20.523289","exception":false,"start_time":"2022-05-01T14:22:20.29375","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-07-15T18:52:41.711131Z","iopub.execute_input":"2022-07-15T18:52:41.711798Z","iopub.status.idle":"2022-07-15T18:52:42.994631Z","shell.execute_reply.started":"2022-07-15T18:52:41.711763Z","shell.execute_reply":"2022-07-15T18:52:42.993287Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Start Training","metadata":{"papermill":{"duration":0.065997,"end_time":"2022-05-01T14:22:20.65523","exception":false,"start_time":"2022-05-01T14:22:20.589233","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# For training Unet Model with folds!=0\nfor fold in range(1):\n    print(f'#'*35)\n    print(f'######### Fold: {fold}')\n    print(f'#'*35)\n    train_loader, valid_loader = prepare_loaders(fold=fold, debug=CFG.debug)\n    model     = build_model()\n    optimizer = optim.AdamW(model.parameters(), lr=CFG.lr, weight_decay=CFG.wd)\n    scheduler = fetch_scheduler(optimizer)\n    model, history = run_training(model, optimizer, scheduler,\n                                  device=CFG.device,\n                                  num_epochs=CFG.epochs)","metadata":{"execution":{"iopub.status.busy":"2022-07-13T10:32:00.820057Z","iopub.execute_input":"2022-07-13T10:32:00.820434Z","iopub.status.idle":"2022-07-13T12:20:43.27497Z","shell.execute_reply.started":"2022-07-13T10:32:00.820403Z","shell.execute_reply":"2022-07-13T12:20:43.272881Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"only focaloss, best score - 0.7077 (jaccard) 20 epochs\n\nwithout focalloss , default losses , epoch 10 , best score - 0.7291, effnetb0\n\nwithout focalloss , default losses , epoch 10 , best score - 0.7234, resnet18\n\nfocalloss , default losses , epoch 10 , best score - 0.7027, resnet18\n\ndefault losses , epoch 10 , best score - 0.6944 , deeplabv3 + resnet50 \n\n0.5 ,0.5 default losses , data augmentation change, lr 0.002 , best score - 0.7158, unet + resnet50, epoch 5\n(Valid Dice: 0.7910 | Valid Jaccard: 0.7158 | Train Dice: 0.8462 | Train Jaccard: 0.7793)\n\nLatest update : Valid Dice: 0.7764 | Valid Jaccard: 0.7007 | Train Dice: 0.8764 | Train Jaccard: 0.8156\nTraining complete in 0h 45m 20s\nBest Score: 0.7078\n\n0.5*Bce + 0.5 * dice , epoch 1 , folds 5 , val jaccard : 0.7521 , val_dice: 0.8251\n\n0.4*Bce + 0.4 * dice+0.2 * jaccloss , epoch 1 , folds 5 , val jaccard : 0.7521 , val_dice: 0.8277\n\n0.2*Bce + 0.2 * dice+ 0.4 * JaccardLoss(y_pred,y_true) + 0.2 * TverskyLoss(y_pred, y_true)\n\n0.5 * JaccardLoss(y_pred,y_true) + 0.5 * TverskyLoss(y_pred, y_true) , best score: 0.7685","metadata":{}},{"cell_type":"markdown","source":"# Loss, Dice,Jaccard Curves","metadata":{}},{"cell_type":"code","source":"# PLOT TRAINING\nplt.figure(figsize=(15,5))\nplt.plot(range(CFG.epochs),history['Valid Dice'],label='Valid Dice')\nplt.plot(range(CFG.epochs),history['Valid Jaccard'],label='Valid Jaccard')\nplt.title('Dice & Jaccard'); plt.xlabel('Epoch'); plt.ylabel('');plt.legend(); \nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-11T10:13:45.069148Z","iopub.status.idle":"2022-07-11T10:13:45.069839Z","shell.execute_reply.started":"2022-07-11T10:13:45.069589Z","shell.execute_reply":"2022-07-11T10:13:45.069613Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# PLOT TRAINING\nplt.figure(figsize=(15,5))\nplt.plot(range(CFG.epochs),history['Train Loss'],label='Train Loss')\nplt.plot(range(CFG.epochs),history['Valid Loss'],label='Valid Loss')\nplt.title('LOSS'); plt.xlabel('Epoch'); plt.ylabel('loss');plt.legend(); \nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-11T10:13:45.071109Z","iopub.status.idle":"2022-07-11T10:13:45.071795Z","shell.execute_reply.started":"2022-07-11T10:13:45.071548Z","shell.execute_reply":"2022-07-11T10:13:45.071571Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### For inference and predictions , download the models from the kaggle working directory and then import them in the inference notebook as dataset.","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}