{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 0. Objective","metadata":{}},{"cell_type":"markdown","source":"The objective of this notebook is to provide a minimum viable pipeline using [fastai](https://docs.fast.ai/) for classification and detection of opacity in xray images. Anyone who is new to this competition can start from this notebook and learn the followings:\n\n* Basic understanding of the nature of medical imaging data\n* Introduction to fastai library for [medical imaging](https://docs.fast.ai/medical.imaging)\n* Submission of predictions\n\nThe progress is as following:\n\n    1. Data Overview           (Done)\n    2. Data Preprocessing      (Done)\n    3. Preparing Datablock     (Done)\n    4. Training                (Done)\n    5. Predictions             (tbd, predict() not yet available)\n    6. Subsmission             (tdb)\n\nI am using following 2 notebooks as reference:\n* https://www.kaggle.com/avirdee/siim-covid-19-initial-pipeline-fastai\n* https://github.com/muellerzr/Practical-Deep-Learning-for-Coders-2.0/blob/master/Computer%20Vision/06_Object_Detection.ipynb\n\n    ","metadata":{}},{"cell_type":"markdown","source":"# 1. Data Overview","metadata":{}},{"cell_type":"code","source":"# Load Grassroots DICOM (GDCM) for xray DICOM files\n!pip install python-gdcm -q\n\n# Load glob2\n!pip install glob2\n\n# Load tqdm\n!pip install tqdm","metadata":{"execution":{"iopub.status.busy":"2021-06-28T14:44:08.028278Z","iopub.execute_input":"2021-06-28T14:44:08.028612Z","iopub.status.idle":"2021-06-28T14:44:29.726112Z","shell.execute_reply.started":"2021-06-28T14:44:08.028582Z","shell.execute_reply":"2021-06-28T14:44:29.725154Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Loading necessary packages\nimport os\nfrom datetime import datetime\nimport pandas as pd\nimport numpy as np\nimport glob2\nfrom tqdm.notebook import tqdm\nimport cv2\nimport gdcm\nimport pydicom\nfrom pydicom.pixel_data_handlers.util import apply_voi_lut\nfrom fastai.vision.all import *\nfrom fastai.medical.imaging import *\nfrom torchvision.utils import save_image","metadata":{"execution":{"iopub.status.busy":"2021-06-28T14:44:29.727874Z","iopub.execute_input":"2021-06-28T14:44:29.728221Z","iopub.status.idle":"2021-06-28T14:44:33.545281Z","shell.execute_reply.started":"2021-06-28T14:44:29.728182Z","shell.execute_reply":"2021-06-28T14:44:33.544401Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"SOURCE = '/kaggle/input/siim-covid19-detection'\nos.listdir(SOURCE)","metadata":{"execution":{"iopub.status.busy":"2021-06-28T14:44:33.547078Z","iopub.execute_input":"2021-06-28T14:44:33.547419Z","iopub.status.idle":"2021-06-28T14:44:33.556856Z","shell.execute_reply.started":"2021-06-28T14:44:33.547374Z","shell.execute_reply":"2021-06-28T14:44:33.555905Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_image_level = pd.read_csv(f'{SOURCE}/train_image_level.csv')\ntrain_study_level = pd.read_csv(f'{SOURCE}/train_study_level.csv')\nsample_submission = pd.read_csv(f'{SOURCE}/sample_submission.csv')","metadata":{"execution":{"iopub.status.busy":"2021-06-28T14:44:33.558634Z","iopub.execute_input":"2021-06-28T14:44:33.559011Z","iopub.status.idle":"2021-06-28T14:44:33.626175Z","shell.execute_reply.started":"2021-06-28T14:44:33.558975Z","shell.execute_reply":"2021-06-28T14:44:33.625331Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_image_level.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-28T14:44:33.627376Z","iopub.execute_input":"2021-06-28T14:44:33.627717Z","iopub.status.idle":"2021-06-28T14:44:33.649673Z","shell.execute_reply.started":"2021-06-28T14:44:33.627682Z","shell.execute_reply":"2021-06-28T14:44:33.648618Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_study_level.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-28T14:44:33.650965Z","iopub.execute_input":"2021-06-28T14:44:33.651306Z","iopub.status.idle":"2021-06-28T14:44:33.661062Z","shell.execute_reply.started":"2021-06-28T14:44:33.651271Z","shell.execute_reply":"2021-06-28T14:44:33.660061Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_submission.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-28T14:44:33.662513Z","iopub.execute_input":"2021-06-28T14:44:33.663170Z","iopub.status.idle":"2021-06-28T14:44:33.674012Z","shell.execute_reply.started":"2021-06-28T14:44:33.663131Z","shell.execute_reply":"2021-06-28T14:44:33.673156Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# XRAY Files\ndef get_dcm_files(path, recurse=True, folders=None):\n    \"Get image files in `path` recursively, only in `folders`, if specified.\"\n    return get_files(path, extensions=['.dcm'], recurse=recurse, folders=folders)\n\n# Read DICOM files\nTRAIN_DIR = f'{SOURCE}/train/'\nTEST_DIR =  f'{SOURCE}/test/'\ntrain_dcm = get_dcm_files(TRAIN_DIR)\ntest_dcm = get_dcm_files(TEST_DIR)\n\n# Looking on a sample XRAY\nxray_sample = train_dcm[1].dcmread()\nxray_sample","metadata":{"execution":{"iopub.status.busy":"2021-06-28T14:44:33.676795Z","iopub.execute_input":"2021-06-28T14:44:33.677173Z","iopub.status.idle":"2021-06-28T14:45:04.692347Z","shell.execute_reply.started":"2021-06-28T14:44:33.677137Z","shell.execute_reply":"2021-06-28T14:45:04.691395Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"xray_sample.show()","metadata":{"execution":{"iopub.status.busy":"2021-06-28T14:45:04.696748Z","iopub.execute_input":"2021-06-28T14:45:04.698816Z","iopub.status.idle":"2021-06-28T14:45:07.549485Z","shell.execute_reply.started":"2021-06-28T14:45:04.698773Z","shell.execute_reply":"2021-06-28T14:45:07.548625Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2. Data Preprocessing","metadata":{}},{"cell_type":"code","source":"# Merging study_level and image_level\n# rename id column in study_level to StudyInstanceUID\ntrain_study_level.rename(columns = {'id':'StudyInstanceUID'}, inplace = True)\n\n# remove _study from StudyInstanceUID\ntrain_study_level['StudyInstanceUID'] = train_study_level['StudyInstanceUID'].str.replace('_study', '')\n\n# merge\ndf_train = pd.merge(train_image_level, train_study_level, on='StudyInstanceUID')\n\n# remove _image from id column\ndf_train['id'] = df_train['id'].str.replace('_image', '')\n\n# rename id column as imageID\ndf_train.rename(columns = {'id':'imageID'}, inplace = True)\n\n# renaming target columns\ndf_train.rename(columns = {'Negative for Pneumonia':'negative'}, inplace = True)\ndf_train.rename(columns = {'Typical Appearance':'typical'}, inplace = True)\ndf_train.rename(columns = {'Indeterminate Appearance':'indeterminate'}, inplace = True)\ndf_train.rename(columns = {'Atypical Appearance':'atypical'}, inplace = True)\n\n# Create a new target column\ncategories = ['negative','typical','indeterminate','atypical']\ndf = df_train[categories]\ndf_train[\"target\"] = pd.Series(df.columns[np.where(df!=0)[1]])\ndf_train.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-28T14:45:07.550601Z","iopub.execute_input":"2021-06-28T14:45:07.550955Z","iopub.status.idle":"2021-06-28T14:45:07.605789Z","shell.execute_reply.started":"2021-06-28T14:45:07.550917Z","shell.execute_reply":"2021-06-28T14:45:07.604719Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Creating path column for each image\nTRAIN_DIR = f'{SOURCE}/train/'\npaths = []\n\nfor instance_id in tqdm(df_train['StudyInstanceUID']):\n    paths.append(glob.glob(os.path.join(TRAIN_DIR, instance_id +\"/*/*\"))[0])\n\ndf_train['path'] = paths\ndf_train[:5]","metadata":{"execution":{"iopub.status.busy":"2021-06-28T14:45:07.607250Z","iopub.execute_input":"2021-06-28T14:45:07.607597Z","iopub.status.idle":"2021-06-28T14:45:12.662379Z","shell.execute_reply.started":"2021-06-28T14:45:07.607561Z","shell.execute_reply":"2021-06-28T14:45:12.661644Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Calculate number of bounding boxes\n# Source: https://www.kaggle.com/avirdee/siim-covid-19-initial-pipeline-fastai\nnum_of_boxes = []\nfor i in df_train.index:\n    val_len = len(df_train['label'][i].split(' '))\n    val = df_train['label'][i].split(' ')\n    label = df_train['target'][i]\n    box_count = val_len//6\n    num_of_boxes.append(box_count)\n    \ndf_train['num_of_boxes'] = num_of_boxes\ndf_train.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-28T14:45:12.663692Z","iopub.execute_input":"2021-06-28T14:45:12.664029Z","iopub.status.idle":"2021-06-28T14:45:12.846461Z","shell.execute_reply.started":"2021-06-28T14:45:12.663994Z","shell.execute_reply":"2021-06-28T14:45:12.845698Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train['num_of_boxes'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2021-06-28T14:45:12.847721Z","iopub.execute_input":"2021-06-28T14:45:12.848063Z","iopub.status.idle":"2021-06-28T14:45:12.855884Z","shell.execute_reply.started":"2021-06-28T14:45:12.848026Z","shell.execute_reply":"2021-06-28T14:45:12.855012Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Parse label column\nbboxes = []\nfor i in df_train.index:\n    num_of_boxes = df_train['num_of_boxes'][i]\n    val = df_train['label'][i].split(' ')\n    if num_of_boxes == 1: boxes = val[2:6]\n    if num_of_boxes == 2: boxes = val[2:6] + val[8:12]\n    if num_of_boxes == 3: boxes = val[2:6] + val[8:12] + val[14:18]\n    if num_of_boxes == 4: boxes = val[2:6] + val[8:12] + val[14:18] + val[20:24]\n    bboxes.append(boxes)\n    \ndf_train['parsed_label'] = bboxes\ndf_train.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-28T14:45:12.857348Z","iopub.execute_input":"2021-06-28T14:45:12.857770Z","iopub.status.idle":"2021-06-28T14:45:13.131369Z","shell.execute_reply.started":"2021-06-28T14:45:12.857730Z","shell.execute_reply":"2021-06-28T14:45:13.130515Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3. Preparing DataBlock","metadata":{}},{"cell_type":"code","source":"# Subsetting df_train on columns required for datablock\ndf_datablock = df_train[['imageID', 'target', 'parsed_label', 'path']].copy()\ndf_datablock.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-28T14:45:13.132604Z","iopub.execute_input":"2021-06-28T14:45:13.132966Z","iopub.status.idle":"2021-06-28T14:45:13.150510Z","shell.execute_reply.started":"2021-06-28T14:45:13.132929Z","shell.execute_reply":"2021-06-28T14:45:13.149665Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Defining get_items() as Path() object to file\nim_df = df_datablock['path'].unique()\nfns = [Path(str(f'{fn}')) for fn in im_df]\n#fns[:5]\n\ndef get_items(noop): return fns","metadata":{"execution":{"iopub.status.busy":"2021-06-28T14:45:13.151761Z","iopub.execute_input":"2021-06-28T14:45:13.152097Z","iopub.status.idle":"2021-06-28T14:45:13.215068Z","shell.execute_reply.started":"2021-06-28T14:45:13.152059Z","shell.execute_reply":"2021-06-28T14:45:13.214155Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Convert data frame to numpy array for faster processing\ndf_np = df_datablock.to_numpy()\ndf_np[0]","metadata":{"execution":{"iopub.status.busy":"2021-06-28T14:45:13.216325Z","iopub.execute_input":"2021-06-28T14:45:13.216821Z","iopub.status.idle":"2021-06-28T14:45:13.222465Z","shell.execute_reply.started":"2021-06-28T14:45:13.216782Z","shell.execute_reply":"2021-06-28T14:45:13.221641Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_tmp_bbox(fn):\n    rows = np.where(df_np[:,0] == fn.name[:-4])\n    bboxs = df_np[rows][:,-2][0]\n    return np.array([np.fromstring(b, sep=',') for b in bboxs])\n\ndef get_tmp_lbl(fn):\n    rows = np.where((df_np[:, 0] == fn.name[:-4]))\n    bboxs = len(df_np[rows][:,-2][0])\n    if bboxs > 12:\n        return np.concatenate(([df_np[rows][:,1]]*4))\n    if bboxs > 8:\n        return np.concatenate(([df_np[rows][:,1]]*3))\n    if bboxs > 4:\n        return np.concatenate(([df_np[rows][:,1]]*2))\n    else:\n        return df_np[rows][:,1]","metadata":{"execution":{"iopub.status.busy":"2021-06-28T14:45:13.223645Z","iopub.execute_input":"2021-06-28T14:45:13.224155Z","iopub.status.idle":"2021-06-28T14:45:13.233815Z","shell.execute_reply.started":"2021-06-28T14:45:13.224118Z","shell.execute_reply":"2021-06-28T14:45:13.233063Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"get_tmp_bbox(get_items(SOURCE)[2])","metadata":{"execution":{"iopub.status.busy":"2021-06-28T14:45:13.234905Z","iopub.execute_input":"2021-06-28T14:45:13.235395Z","iopub.status.idle":"2021-06-28T14:45:13.249335Z","shell.execute_reply.started":"2021-06-28T14:45:13.235358Z","shell.execute_reply":"2021-06-28T14:45:13.248132Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"get_tmp_lbl(get_items(SOURCE)[2])","metadata":{"execution":{"iopub.status.busy":"2021-06-28T14:45:13.251094Z","iopub.execute_input":"2021-06-28T14:45:13.251783Z","iopub.status.idle":"2021-06-28T14:45:13.260319Z","shell.execute_reply.started":"2021-06-28T14:45:13.251704Z","shell.execute_reply":"2021-06-28T14:45:13.259316Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"bboxs = get_tmp_bbox(fns[0])\nlbls = get_tmp_lbl(fns[0])\narr = np.array([fns[0].name[:-4], bboxs, lbls], dtype=object)\narr","metadata":{"execution":{"iopub.status.busy":"2021-06-28T14:45:13.261805Z","iopub.execute_input":"2021-06-28T14:45:13.262209Z","iopub.status.idle":"2021-06-28T14:45:13.274920Z","shell.execute_reply.started":"2021-06-28T14:45:13.262131Z","shell.execute_reply":"2021-06-28T14:45:13.273776Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Whole dataset\nfor path in fns[1:]:\n    bbox = get_tmp_bbox(path)\n    lbl = get_tmp_lbl(path)\n    arr2 = np.array([path.name[:-4], bbox, lbl], dtype='object')\n    arr = np.vstack((arr, arr2))","metadata":{"execution":{"iopub.status.busy":"2021-06-28T14:45:13.276708Z","iopub.execute_input":"2021-06-28T14:45:13.277232Z","iopub.status.idle":"2021-06-28T14:45:17.632957Z","shell.execute_reply.started":"2021-06-28T14:45:13.277157Z","shell.execute_reply":"2021-06-28T14:45:17.631998Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_bbox(fn):\n    idx = np.where((arr[:,0] == fn.name[:-4]))\n    return arr[idx][0][1]\n\ndef get_lbl(fn):\n    idx = np.where((arr[:,0] == fn.name[:-4]))\n    return arr[idx][0][-1]","metadata":{"execution":{"iopub.status.busy":"2021-06-28T14:45:17.636418Z","iopub.execute_input":"2021-06-28T14:45:17.636714Z","iopub.status.idle":"2021-06-28T14:45:17.641520Z","shell.execute_reply.started":"2021-06-28T14:45:17.636686Z","shell.execute_reply":"2021-06-28T14:45:17.640679Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"get_bbox(get_items(SOURCE)[2])","metadata":{"execution":{"iopub.status.busy":"2021-06-28T14:45:17.643154Z","iopub.execute_input":"2021-06-28T14:45:17.643743Z","iopub.status.idle":"2021-06-28T14:45:17.660091Z","shell.execute_reply.started":"2021-06-28T14:45:17.643705Z","shell.execute_reply":"2021-06-28T14:45:17.658865Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"get_lbl(get_items(SOURCE)[2])","metadata":{"execution":{"iopub.status.busy":"2021-06-28T14:45:17.661483Z","iopub.execute_input":"2021-06-28T14:45:17.661876Z","iopub.status.idle":"2021-06-28T14:45:17.671068Z","shell.execute_reply.started":"2021-06-28T14:45:17.661839Z","shell.execute_reply":"2021-06-28T14:45:17.670064Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Source: https://www.kaggle.com/avirdee/siim-covid-19-initial-pipeline-fastai\nclass HistView(PILDicom):\n    \"View histogram scaled version of the pixel array\"\n    @classmethod\n    def create(cls, fn:(Path, str, bytes))->None:\n        if isinstance(fn, bytes): im = pydicom.dcmread(pydicom.filebase.DicomBytesIO(fn))\n        if isinstance(fn, (Path, str)): im = pydicom.dcmread(fn)\n        scaled = np.array(im.hist_scaled())\n        scaled = scaled - np.min(scaled)\n        scaled = scaled / np.max(scaled)\n        scaled = (scaled * 255).astype(np.uint8)\n        pill_im = Image.fromarray(scaled)\n        return cls(pill_im)","metadata":{"execution":{"iopub.status.busy":"2021-06-28T14:45:17.672605Z","iopub.execute_input":"2021-06-28T14:45:17.673328Z","iopub.status.idle":"2021-06-28T14:45:17.682440Z","shell.execute_reply.started":"2021-06-28T14:45:17.673286Z","shell.execute_reply":"2021-06-28T14:45:17.681656Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"set_seed(7)\ndatablock = DataBlock(blocks=(ImageBlock(cls=HistView), BBoxBlock, BBoxLblBlock),\n                 get_items=get_items,\n                 splitter=RandomSplitter(),\n                 get_y=[get_bbox, get_lbl],\n                 item_tfms=[Resize(128, method='pad'),],\n                 batch_tfms=[Rotate(), Flip(), Dihedral(), Normalize.from_stats(*imagenet_stats)],\n                 n_inp=1)\n\ndls = datablock.dataloaders(TRAIN_DIR, bs=128)\ndls.show_batch(max_n=20, ncols=5)","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-06-28T14:45:27.200553Z","iopub.execute_input":"2021-06-28T14:45:27.200958Z","iopub.status.idle":"2021-06-28T14:48:48.937832Z","shell.execute_reply.started":"2021-06-28T14:45:27.200926Z","shell.execute_reply":"2021-06-28T14:48:48.936992Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Checking shape of a batch\nbatch = dls.one_batch()\nbatch[0].shape","metadata":{"execution":{"iopub.status.busy":"2021-06-28T14:48:48.939308Z","iopub.execute_input":"2021-06-28T14:48:48.939894Z","iopub.status.idle":"2021-06-28T14:51:59.364342Z","shell.execute_reply.started":"2021-06-28T14:48:48.939853Z","shell.execute_reply":"2021-06-28T14:51:59.363532Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"batch[1].shape","metadata":{"execution":{"iopub.status.busy":"2021-06-28T14:51:59.365961Z","iopub.execute_input":"2021-06-28T14:51:59.366215Z","iopub.status.idle":"2021-06-28T14:51:59.372960Z","shell.execute_reply.started":"2021-06-28T14:51:59.366190Z","shell.execute_reply":"2021-06-28T14:51:59.371929Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"batch[1][0]","metadata":{"execution":{"iopub.status.busy":"2021-06-28T14:51:59.374577Z","iopub.execute_input":"2021-06-28T14:51:59.374971Z","iopub.status.idle":"2021-06-28T14:51:59.386906Z","shell.execute_reply.started":"2021-06-28T14:51:59.374933Z","shell.execute_reply":"2021-06-28T14:51:59.386132Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"batch[2].shape","metadata":{"execution":{"iopub.status.busy":"2021-06-28T14:51:59.388200Z","iopub.execute_input":"2021-06-28T14:51:59.388536Z","iopub.status.idle":"2021-06-28T14:51:59.395688Z","shell.execute_reply.started":"2021-06-28T14:51:59.388502Z","shell.execute_reply":"2021-06-28T14:51:59.394667Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"batch[2][0]","metadata":{"execution":{"iopub.status.busy":"2021-06-28T14:51:59.397081Z","iopub.execute_input":"2021-06-28T14:51:59.397578Z","iopub.status.idle":"2021-06-28T14:51:59.407072Z","shell.execute_reply.started":"2021-06-28T14:51:59.397539Z","shell.execute_reply":"2021-06-28T14:51:59.406068Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4. Training","metadata":{}},{"cell_type":"code","source":"# Source: https://github.com/muellerzr/Practical-Deep-Learning-for-Coders-2.0/blob/master/Computer%20Vision/06_Object_Detection.ipynb\n!git clone https://github.com/muellerzr/Practical-Deep-Learning-for-Coders-2.0.git\n%cd \"Practical-Deep-Learning-for-Coders-2.0/Computer Vision\"","metadata":{"execution":{"iopub.status.busy":"2021-06-28T10:08:06.724806Z","iopub.execute_input":"2021-06-28T10:08:06.725135Z","iopub.status.idle":"2021-06-28T10:08:13.753509Z","shell.execute_reply.started":"2021-06-28T10:08:06.725103Z","shell.execute_reply":"2021-06-28T10:08:13.752576Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from imports import *","metadata":{"execution":{"iopub.status.busy":"2021-06-28T10:08:13.755183Z","iopub.execute_input":"2021-06-28T10:08:13.755557Z","iopub.status.idle":"2021-06-28T10:08:13.770399Z","shell.execute_reply.started":"2021-06-28T10:08:13.75552Z","shell.execute_reply":"2021-06-28T10:08:13.76963Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"encoder = create_body(resnet34, pretrained=True)","metadata":{"execution":{"iopub.status.busy":"2021-06-28T10:08:13.774101Z","iopub.execute_input":"2021-06-28T10:08:13.774373Z","iopub.status.idle":"2021-06-28T10:08:20.222678Z","shell.execute_reply.started":"2021-06-28T10:08:13.77432Z","shell.execute_reply":"2021-06-28T10:08:20.221714Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"arch = RetinaNet(encoder, get_c(dls), final_bias=-4)","metadata":{"execution":{"iopub.status.busy":"2021-06-28T10:08:20.224252Z","iopub.execute_input":"2021-06-28T10:08:20.224751Z","iopub.status.idle":"2021-06-28T10:08:20.599872Z","shell.execute_reply.started":"2021-06-28T10:08:20.224715Z","shell.execute_reply":"2021-06-28T10:08:20.598967Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Loss function\nratios = [1/2,1,2]\nscales = [1,2**(-1/3), 2**(-2/3)]","metadata":{"execution":{"iopub.status.busy":"2021-06-28T10:08:20.601127Z","iopub.execute_input":"2021-06-28T10:08:20.601515Z","iopub.status.idle":"2021-06-28T10:08:20.608103Z","shell.execute_reply.started":"2021-06-28T10:08:20.601478Z","shell.execute_reply":"2021-06-28T10:08:20.607281Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"crit = RetinaNetFocalLoss(scales=scales, ratios=ratios)","metadata":{"execution":{"iopub.status.busy":"2021-06-28T10:08:20.609364Z","iopub.execute_input":"2021-06-28T10:08:20.609748Z","iopub.status.idle":"2021-06-28T10:08:20.617456Z","shell.execute_reply.started":"2021-06-28T10:08:20.609707Z","shell.execute_reply":"2021-06-28T10:08:20.616684Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def _retinanet_split(m): \n    return L(m.encoder,nn.Sequential(m.c5top6, m.p6top7, m.merges, m.smoothers, m.classifier, m.box_regressor)).map(params)","metadata":{"execution":{"iopub.status.busy":"2021-06-28T10:08:20.618709Z","iopub.execute_input":"2021-06-28T10:08:20.619129Z","iopub.status.idle":"2021-06-28T10:08:20.626744Z","shell.execute_reply.started":"2021-06-28T10:08:20.619095Z","shell.execute_reply":"2021-06-28T10:08:20.625929Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learn = Learner(dls, arch, loss_func=crit, splitter=_retinanet_split)","metadata":{"execution":{"iopub.status.busy":"2021-06-28T10:08:20.629288Z","iopub.execute_input":"2021-06-28T10:08:20.629652Z","iopub.status.idle":"2021-06-28T10:08:20.637614Z","shell.execute_reply.started":"2021-06-28T10:08:20.629618Z","shell.execute_reply":"2021-06-28T10:08:20.636851Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learn.freeze()","metadata":{"execution":{"iopub.status.busy":"2021-06-28T10:08:20.639385Z","iopub.execute_input":"2021-06-28T10:08:20.639968Z","iopub.status.idle":"2021-06-28T10:08:20.663014Z","shell.execute_reply.started":"2021-06-28T10:08:20.639932Z","shell.execute_reply":"2021-06-28T10:08:20.662242Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learn.fit_one_cycle(2, slice(1e-5, 1e-4))","metadata":{"execution":{"iopub.status.busy":"2021-06-28T10:08:20.664462Z","iopub.execute_input":"2021-06-28T10:08:20.664881Z","iopub.status.idle":"2021-06-28T13:10:10.514845Z","shell.execute_reply.started":"2021-06-28T10:08:20.664844Z","shell.execute_reply":"2021-06-28T13:10:10.513948Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Saving weights of the trained model\nTRAINED_MODELS_DIR = '/kaggle/working/trained_models_dir/'\nos.mkdir(TRAINED_MODELS_DIR)\n\ntimestamp = datetime.now().strftime(\"_%Y%m%d_%H%M%S_\")\nfile_name = TRAINED_MODELS_DIR + \"trainedModelWeights\" + timestamp\nlearn.save(file = file_name)","metadata":{"execution":{"iopub.status.busy":"2021-06-28T13:56:30.064452Z","iopub.execute_input":"2021-06-28T13:56:30.064795Z","iopub.status.idle":"2021-06-28T13:56:30.405087Z","shell.execute_reply.started":"2021-06-28T13:56:30.064765Z","shell.execute_reply":"2021-06-28T13:56:30.403909Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Exporting the trained model\ntimestamp = datetime.now().strftime(\"_%Y%m%d_%H%M%S_\")\nfile_name = TRAINED_MODELS_DIR + \"trainedModelExport\" + timestamp + \".pkl\"\nlearn.export(fname = file_name)\nos.listdir(TRAINED_MODELS_DIR)","metadata":{"execution":{"iopub.status.busy":"2021-06-28T13:56:32.955283Z","iopub.execute_input":"2021-06-28T13:56:32.955664Z","iopub.status.idle":"2021-06-28T13:56:33.226451Z","shell.execute_reply.started":"2021-06-28T13:56:32.955632Z","shell.execute_reply":"2021-06-28T13:56:33.225464Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 5. Predictions","metadata":{}},{"cell_type":"code","source":"sample_img_path = TEST_DIR + '/2fb11712bc93/b056067b8455/a29c5a68b07b.dcm'\nsample_img_path","metadata":{"execution":{"iopub.status.busy":"2021-06-28T14:06:10.751452Z","iopub.execute_input":"2021-06-28T14:06:10.751783Z","iopub.status.idle":"2021-06-28T14:06:10.758844Z","shell.execute_reply.started":"2021-06-28T14:06:10.751755Z","shell.execute_reply":"2021-06-28T14:06:10.757864Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learn.predict(sample_img_path)","metadata":{"execution":{"iopub.status.busy":"2021-06-28T14:07:39.239874Z","iopub.execute_input":"2021-06-28T14:07:39.240251Z","iopub.status.idle":"2021-06-28T14:07:40.662678Z","shell.execute_reply.started":"2021-06-28T14:07:39.240215Z","shell.execute_reply":"2021-06-28T14:07:40.658437Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 6. Submission","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}