{"cells":[{"metadata":{},"cell_type":"markdown","source":"# RANZCR Tracheal Bifurcations Datasets\n\nThis notebook uses two Datasets available for RANZCR - \n*     [raddar's bifurcation location predictions](https://www.kaggle.com/raddar/ranzcr-clip-tracheal-bifurcation)\n*     [dr konya's 5k manual annotations](https://www.kaggle.com/sandorkonya/5k-trachea-bifurcation-on-chest-xray)"},{"metadata":{},"cell_type":"markdown","source":"CSV format of 5K tracheal bifurcations annotation is created from json with predicted locations added and saved as output\n\nETT Visualisations are included for examples of RANZCR annotation intubation and the two tracheal bifurcations from datasets"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport os\n\nimport ast\nimport cv2\nimport matplotlib.pyplot as plt\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/ranzcr-clip-catheter-line-classification/train.csv')\ntrain_annotations = pd.read_csv('../input/ranzcr-clip-catheter-line-classification/train_annotations.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"trncols = train.columns.values\ntarget_cols = trncols[1:-1]  # Target Columns from train","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"https://www.kaggle.com/raddar/ranzcr-clip-tracheal-bifurcation \n\nThis dataset contains tracheal (bronchial) bifurcation location predictions of a YOLOv3 detector trained on several thousand hand labelled images from external data sources. Bifurcation points are used as a reference for deciding if intubation tube has been inserted correctly. Intubation is considered normal when tube tip is no less than 3cm above bifurcation point. If the distance is lower - abnormality is considered."},{"metadata":{"trusted":true},"cell_type":"code","source":"tbif = pd.read_csv('../input/ranzcr-clip-tracheal-bifurcation/RANZCR_CLiP_tracheal_bifurcation.csv')\nlen(tbif)  # all entries in train","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tbif.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"https://www.kaggle.com/sandorkonya/5k-trachea-bifurcation-on-chest-xray\n\nThe dataset contains manually annotated 5281 trachea bifurcation on x-rays of the dataset of the current challenge.    \n"},{"metadata":{"trusted":true},"cell_type":"code","source":"tbif5k = pd.read_json('../input/5k-trachea-bifurcation-on-chest-xray/trachea_annotations.json', orient='index')\ntbif5k.reset_index(drop=True, inplace=True)\nlen(tbif5k)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# add StudyInstanceUID from jpg filename\ntbif5k['StudyInstanceUID'] = tbif5k['filename'].apply(lambda x: x.split('.jpg')[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_point_xy(x):\n    cx = 0\n    cy = 0\n    # skip empty regions\n    if len(x)>=1:\n        xd = x[0]  # dict from regions   \n        cx = xd['shape_attributes']['cx'] \n        cy = xd['shape_attributes']['cy']\n   \n    return pd.Series([cx,cy])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# note 2 entries with empty regions - set cx cy columns as 0s default\ntbif5k['cx'] = 0\ntbif5k['cy'] = 0\n\ntbif5k[['cx', 'cy']] = tbif5k.apply(lambda row: get_point_xy(row.regions), axis=1)\ntbif5k.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Copy of 5K trachea bifurcation for output as csv. Drop columns not needed.\n\nAdd raddar's trachea bifurcation predictions for comparison"},{"metadata":{"trusted":true},"cell_type":"code","source":"tbif5kout = tbif5k.copy()\ntbif5kout.drop(['filename', 'size', 'regions', 'file_attributes'], axis=1, inplace=True)\ntbif5kout = pd.merge(tbif5kout, tbif, on=['StudyInstanceUID'], how='left')\ntbif5kout.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Output 5K tracheal bifurcations annotation points with dataset predicted included"},{"metadata":{"trusted":true},"cell_type":"code","source":"tbif5kout.to_csv('ranzcr_5K_tracheal_bifurcation_annotations.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Merge 5K with train to get target colums for visualisations"},{"metadata":{"trusted":true},"cell_type":"code","source":"tbif5kout = pd.merge(tbif5kout, train, on=['StudyInstanceUID'], how='left')\ntbif5kout.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def f_table(list1):\n    table_dic = {}\n    for i in list1:\n        if i not in table_dic.keys():\n            table_dic[i] = 1\n        else:\n            table_dic[i] += 1\n    return(table_dic)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Determine what Targets are present in 5K dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"tar_freq = np.array([np.min(list(f_table(tbif5kout[target_cols].iloc[:,i]).values())) for i in range(len(target_cols))])\ntbif5ktarg = pd.DataFrame(\n                {                \n                'target' : target_cols,\n                'count' : tar_freq,                     \n                })    \ntbif5ktarg.head(11)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# ETT UIDs in 5K\nabnorm5k_uid = tbif5kout.loc[tbif5kout['ETT - Abnormal']==1,'StudyInstanceUID'].tolist()\nborder5k_uid = tbif5kout.loc[tbif5kout['ETT - Borderline']==1, 'StudyInstanceUID'].tolist()  \nnorm5k_uid   = tbif5kout.loc[tbif5kout['ETT - Normal']==1,'StudyInstanceUID'].tolist()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# ETT UIDS in annotations for borderline\nborderann_uid = train_annotations.loc[train_annotations.label=='ETT - Borderline', 'StudyInstanceUID'].tolist() \nbordset = (set(borderann_uid).intersection(set(border5k_uid)))\nborderlist = list(bordset)\nlen(borderlist)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# ETT UIDS in annotations for Normal\nnormann_uid = train_annotations.loc[train_annotations.label=='ETT - Normal', 'StudyInstanceUID'].tolist() \nnormset = (set(normann_uid).intersection(set(norm5k_uid)))\nnormlist = list(normset)\nlen(normlist)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Tracheal Bifurcations Visualisation for ETT"},{"metadata":{"trusted":true},"cell_type":"code","source":"# ref https://www.kaggle.com/raddar/simple-ett-bifurcation-visualization\ndef plot_xray(StudyInstanceUID, label):\n    \"\"\"\n    intubation as green (if annotation exists)\n    bifurcation as red (from raddar's predicted tracheal bifurcation)\n    bifurc 5k as blue (from 5k annotations)\n    \"\"\"\n    has_annot = len(train_annotations.loc[(train_annotations.StudyInstanceUID==StudyInstanceUID) & (train_annotations.label==label)] )\n    img = cv2.imread('../input/ranzcr-clip-catheter-line-classification/train/'+StudyInstanceUID+'.jpg')\n    bifurc_5k = (tbif5kout.loc[tbif5kout.StudyInstanceUID==StudyInstanceUID,['cx', 'cy']].values[0])\n    bifurcation = ast.literal_eval(tbif5kout.loc[tbif5kout.StudyInstanceUID==StudyInstanceUID,'tracheal_bifurcation'].values[0])\n    if has_annot > 0:\n        intubation = ast.literal_eval(train_annotations.loc[(train_annotations.StudyInstanceUID==StudyInstanceUID) & (train_annotations.label==label),'data'].values[0])[0]\n        img = cv2.circle(img, tuple(intubation), 50, (0,255,0), 10)\n    img = cv2.circle(img, tuple(bifurcation), 50, (255,0,0), 10)        \n    img = cv2.circle(img,(bifurc_5k[0], bifurc_5k[1]),50,(0,0,255), 10)  \n    \n    plt.figure(figsize=(12,12))\n    plt.title(label = (StudyInstanceUID + '   ' + label))\n    \n    plt.imshow(img)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Plot some examples for ETT - Abnormal \n\nIntubation is green if annotation exists. Bifurcation blue for 5K annotations and red for raddar's predicted \n"},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_xray(abnorm5k_uid[1],'ETT - Abnormal' )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_xray(abnorm5k_uid[5],'ETT - Abnormal' )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Plot some examples for ETT - Borderline"},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_xray(borderlist[30],'ETT - Borderline' )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_xray(borderlist[55],'ETT - Borderline' )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Plot some examples for ETT - Normal"},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_xray(normlist[100],'ETT - Normal' )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_xray(normlist[400],'ETT - Normal' )","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}