{"cells":[{"metadata":{},"cell_type":"markdown","source":"Let's try using a transfer learning model that was specifically **trained on chest x-ray images! **\n\n#### CheXNet - Keras\n\n* CheXNet is based on Densenet 121, which was itself pretrained on imagenet, before being finetuned on ChestX-ray14, which contained 112,120 frontal view greyscale X-rays from 30,805 patients. \n    * For more about CheXnet, check out the original article or github with the trained model: https://github.com/brucechou1983/CheXNet-Keras\n* Loading the model naively won't work, but I provide a workaround here.\n* Keras - for ease of use! :) \n    \n* Data loading code copied from the kernel [Baseline: Transfer Learning+RandomForest](https://www.kaggle.com/titericz/baseline-transfer-learning-randomforest-gpu/) \n* Transfer learning best practices applied - frozen base model and tuning of the output layer, followed by unfreezing all layers and gentler finetuning.\n    * Removing the added dense layer at the end may improve things (just be sure to handle the logits)\n* Note that this is just a starter kernel - there's lots more that could be done to improve the model, the transfer learning, etc' \n* In this initial simple notebook we'll just use chexnet as a static feature extractor, and see how it does vs imagenet pretrained models (.736)\n    * Note - the R.Forest model used assumes (erroneously) that this is multiclass, whilest it is actually a multilabel problem!\n    \nV3: Fixed global pooling to take average pooling of last convolutional block, instead of dense layer. Modified densenet extraction. Added controllable fast run settings. "},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport gc\nimport cv2\nimport cuml\nimport glob\nimport numpy as np\nimport pandas as pd\nfrom numba import cuda\nfrom tqdm.notebook import tqdm\nimport matplotlib.pyplot as plt\nfrom sklearn.metrics import roc_auc_score\nfrom sklearn.ensemble import RandomForestClassifier\nimport tensorflow as tf\nfrom tensorflow.keras import Sequential\nfrom tensorflow.keras.utils import plot_model\nfrom tensorflow.keras.applications import densenet\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.layers import GlobalAveragePooling2D, GlobalMaxPooling2D, AveragePooling2D","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"NUM_CLASSES = 11 # number of target/output classes\n\nIMG_SIZE =  512# 224 # 224 for imagenet , 512 - alt reshape\nIMG_CHANNELS = 3 \n\n## size of the pooled output layer from the model\nPOOLED_OUTPUT_SIZE = 1024 # 1024 for densenet 121, 2048 for mobilenet? \n\nchexnet_weights_path = \"../input/chexnet-keras-weights/brucechou1983_CheXNet_Keras_0.3.0_weights.h5\"\n\nFAST_RUN = False # use only a few rows, for fast debugging\nFAST_RUN_SAMPLES = 150 # num rows to use from train, test when in fast_mode","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"### if you don't want to depend on kaggle datasets, download the CheXNet weights. You cannot simply load them as inbuilt weights, as in TF!\n\n# !wget --no-check-certificate \\\n#         \"https://storage.googleapis.com/kaggle-datasets/66426/130851/brucechou1983_CheXNet_Keras_0.3.0_weights.h5.zip?GoogleAccessId=web-data@kaggle-161607.iam.gserviceaccount.com&Expires=1561942557&Signature=LgBs0ZzvkJ2Re%2BSuUX1JSq4%2B8DhKC1Ur4evO5L%2F4ArqEzSE2BuRj%2BrfNBOMKedVevNZNr2tuljEzE7frleWdq2yRuim2eRygRcAlpauT1wsfOc9i%2BqE%2BiFLDM03CJWV14cURqf%2FS6h64yCNvTqB%2BywEs2rjKEmZykp%2FWhHVEurINTQp1%2FntTO2rK%2BQMawClqAvo2SVayh4CVNnzzDKeyxm9R0w51FoIL%2BoYQhCVnMJLKk3KeOG8lcreKED5vR7D62KrnJy4ft1Hz2%2BO2pkP0OdDP0QZ4D%2F66bdaN6xi3OJg1g9OizWpkzct3OnLBVuivd344CUKlr25KhRS85JuZ2A%3D%3D\"\\\n#         -O \"/tmp/CheXNet_Keras_0.3.0_weights.h5.zip\"\n\n# local_zip='/tmp/CheXNet_Keras_0.3.0_weights.h5.zip'\n# zip_ref=zipfile.ZipFile(local_zip,'r')\n# zip_ref.extractall('/tmp/CheXNet_Keras_0.3.0_weights.h5')\n# zip_ref.close()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"# build a chexnet model with oretrain weights\nclass chexnet(object):\n    @staticmethod\n    def build(weights_path=chexnet_weights_path, out_size=11, embedding_size=64, activation_type='sigmoid',\n              input_shape=(224, 224, 3),not_frozen=False, embedding_only= False):\n        \n        base_model = densenet.DenseNet121(weights=None,\n                                    include_top=False,\n                                    input_shape=input_shape,\n                                    pooling=\"avg\")\n        ## workaround - add dummy layer then load weights then pop dummy layer, in order to match expected shape for pretrained weights\n        predictions = tf.keras.layers.Dense(14, activation='sigmoid', name='predictions')(base_model.output)\n        base_model = tf.keras.Model(inputs=base_model.input, outputs=predictions)\n        base_model.load_weights(weights_path)\n        base_model.layers.pop()\n        print(\"CheXNet loaded\")\n        \n        base_model.trainable=not_frozen # freeze most layers\n        inputs = tf.keras.Input(shape=input_shape)\n     \n#         if embedding_only:\n            \n        # We make sure that the base_model is running in inference mode here,\n        # by passing `training=False`. This is important for fine-tuning\n        x = base_model(inputs, training=not_frozen) # frozen = freeze layers a bit confusing - double negative\n        # A Dense classifier\n        x = keras.layers.Dropout(0.25)(x)  # Regularize with dropout\n        x = tf.keras.layers.Dense(embedding_size, activation='relu')(x)\n        outputs =  tf.keras.layers.Dense(out_size, activation=activation_type)(x)\n        full_model = tf.keras.Model(inputs, outputs)\n        return full_model\n    \ndef rgb2gray(rgb):\n    return np.dot(rgb[...,:3], [0.2989, 0.5870, 0.1140])\n\n\n## note: it would be more effecient to apply tf.keras.layers.experimental transforms https://www.tensorflow.org/tutorials/images/data_augmentation\n## we could also wrap this inside the data generator, but keep seperate for now .\n## https://www.tensorflow.org/tutorials/images/data_augmentation \n\nimg_aug = ImageDataGenerator(\n    rotation_range=9, width_shift_range=0.1,\n    height_shift_range=0.1,\n    shear_range = 0.05,\n    # ,brightness_range=(0.1,0.9)\n    )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls ../input/ranzcr-clip-catheter-line-classification","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Load train and test as DataFrames"},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/ranzcr-clip-catheter-line-classification/train.csv')\ntest = pd.read_csv('../input/ranzcr-clip-catheter-line-classification/sample_submission.csv')\n\nprint(train.shape)\nprint(test.shape)\nprint(\"# unique patients\",train[\"PatientID\"].nunique())\ntrain.head(7)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Check distribution of labels in train"},{"metadata":{"trusted":true},"cell_type":"code","source":"train.sum(numeric_only = True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.sum(numeric_only = True,axis=1).describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"normal_counts = train[['ETT - Normal','NGT - Normal','CVC - Normal', 'Swan Ganz Catheter Present']].sum(axis=1)\nprint(100*round((normal_counts>0).sum()/train.shape[0],4), \"% of rows have a normal label\")\n\nabnormal_counts = train[['ETT - Abnormal', 'ETT - Borderline', 'NGT - Abnormal', 'NGT - Borderline','NGT - Incompletely Imaged',  'CVC - Abnormal','CVC - Borderline']].sum(axis=1)\nprint(100*round((abnormal_counts>0).sum()/train.shape[0],4), \"% of rows have abnormal labels\")\nprint(\"abnormal counts/labels distribution:\\n\",abnormal_counts.describe())\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"While the problem is multilabel, we see that it's relatively rare for there to be multiple labels in a given case. It's more a case of multiclass, presumably there aren't often multiple catheter/lines/problems simultaenously.\nStill, there are a number of cases with multiple issues/abnormal at once. And we see there's an overlap between being normal for some things and abnormal for others!"},{"metadata":{"trusted":true},"cell_type":"code","source":"train.mean().round(3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Check first image in train"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"img = cv2.imread('../input/ranzcr-clip-catheter-line-classification/train/'+train.StudyInstanceUID.values[0]+'.jpg')\nplt.imshow(img)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import ast\n\nannot = pd.read_csv('../input/ranzcr-clip-catheter-line-classification/train_annotations.csv')\nprint(annot.shape)\nannot.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## If `FAST_RUN` - use only a few rows rows for fast image/data loading & debugging\nif FAST_RUN:\n    train = train.head(FAST_RUN_SAMPLES)\n    test = test.head(FAST_RUN_SAMPLES)\n    annot = annot.head(FAST_RUN_SAMPLES)\n    print(\"Fast run\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Process average of cateter position to be used as a mask."},{"metadata":{"trusted":true},"cell_type":"code","source":"RES = np.zeros( (512,512) )\nfor i in tqdm(range(annot.shape[0])): # ORIG\n# for i in tqdm(range(100)): # fast sample   \n    img = cv2.imread('../input/ranzcr-clip-catheter-line-classification/train/'+annot.StudyInstanceUID.values[i]+'.jpg')\n    img[:] = 0\n    data = eval(annot.data.values[i])\n    for i in range(len(data)-1):\n        img = cv2.line(img, (data[i][1],data[i][0]), (data[i+1][1],data[i+1][0]), (255,255,255), 20 )\n    img = cv2.resize(img,(512,512))\n    RES += img[:,:,0]\n    \nRES /= annot.shape[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.imshow(np.clip(RES,0,1))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"mask = RES.copy()\nmask[mask>0.5] = 1.\nmask[mask<1] = 0\nmask = mask.astype(np.uint8)\nmask = np.stack( (mask,mask,mask), 2 )\n\ndel RES\ngc.collect()\nplt.imshow(mask)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Lets extract features from the images using transfer learning from pretrained Imagenet models."},{"metadata":{"trusted":true},"cell_type":"code","source":"import keras\n# from keras.applications.mobilenet import preprocess_input\nfrom keras.applications.densenet import preprocess_input\n# dir(keras.applications)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# !ls ../input/keras-pretrained-models/","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* Instead of mobilenet or the like, use densenet/chexnet\n    * Note. - model may be improved by scaling colour channels to those expected in imagenet, prior to rgb dummy channels creation"},{"metadata":{"trusted":true},"cell_type":"code","source":"# base = keras.applications.Xception( weights=None,  include_top=True)\n# # Load pretrained imagenet weights\n# base.load_weights('../input/keras-pretrained-models/Xception_Top_ImageNet.h5')\n# base.trainable = False\n# model = keras.Model(inputs=base.input, outputs=base.get_layer('avg_pool').output)\n# model.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# # Instantiate cheXnet model with pretrained weights. Pop last layers, add average pooling\nfrom keras.models import Model\n\nbase = densenet.DenseNet121(weights=None,\n                            include_top=False,\n                            input_shape=(IMG_SIZE,IMG_SIZE,3)\n                           )\n## workaround - add dummy layer then load weights then pop dummy layer, in order to match expected shape for pretrained weights\npredictions = tf.keras.layers.Dense(14, activation='sigmoid', name='predictions')(base.output)\n## ,by_name=True - could save on workaround, but don't know if names will necessarily match + how to validate? - https://github.com/keras-team/keras/issues/5397\nbase = tf.keras.Model(inputs=base.input, outputs=predictions) \nbase.load_weights(chexnet_weights_path)\nprint(\"CheXNet loaded\")\nbase.trainable=False # freeze most layers\nbase.training=False\n\nbase.layers.pop()\n\n### https://stackoverflow.com/questions/41668813/how-to-add-and-remove-new-layers-in-keras-after-loading-weights\nnew_model = GlobalAveragePooling2D()(base.layers[-4].output) \n\nmodel = keras.Model(base.input, new_model)\n# model.summary()\n\n# # model = keras.Model(inputs=base.input, outputs=base.get_layer('avg_pool').output)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.output","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"assert model.output.shape[-1] == POOLED_OUTPUT_SIZE","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Inefficient, but easy to understand for loop to extract features from train images"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_path = '../input/ranzcr-clip-catheter-line-classification/train/'\n\n### original code - learns just static embeddings - we can later try to improve by finetuning our DL model\nemb_train = np.zeros( (train.shape[0],POOLED_OUTPUT_SIZE), dtype=np.float32 )\nfor n, filename in tqdm(enumerate(train.StudyInstanceUID.values), total=train.shape[0]): # ORIG\n    img = cv2.imread(train_path+filename+'.jpg')\n    img = cv2.resize(img,(512,512))\n    img *= mask\n    img = preprocess_input(img)[np.newaxis]\n    emb_train[n] = model.predict(img)[0]\n    \ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Future notebook - retrain the model\n* Load train data\n* Fit on it (in 2 stages for transfer learning mode unfreezing/finetuning) + augment images"},{"metadata":{},"cell_type":"markdown","source":"# Extract features from test images"},{"metadata":{"trusted":true},"cell_type":"code","source":"test_path = '../input/ranzcr-clip-catheter-line-classification/test/'\n\nemb_test = np.zeros( (test.shape[0],POOLED_OUTPUT_SIZE), dtype=np.float32 )\nfor n, filename in tqdm(enumerate(test.StudyInstanceUID.values), total=test.shape[0]): # ORIG\n    img = cv2.imread(test_path+filename+'.jpg')\n    img = cv2.resize(img,(512,512))\n    img *= mask\n    img = preprocess_input(img)[np.newaxis]\n    emb_test[n] = model.predict(img)[0]\n    \ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Delete model and release memory"},{"metadata":{"trusted":true},"cell_type":"code","source":"del model\ngc.collect()\nkeras.backend.clear_session() \ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# I found this trick to clear all Keras allocated memory in GPU."},{"metadata":{"trusted":true},"cell_type":"code","source":"cuda.select_device(0)\ncuda.close()\ncuda.select_device(0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Check labels names"},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()\ntargets = train.columns[1:-1]\nprint(targets)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Split train and valid set: 95%/5%\n\n* Todo: better train/test split - e.g. groupwise split. \n* sklearn crossval predict"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_index = np.where( (np.arange(emb_train.shape[0])%20)!=7 )[0]\nvalid_index = np.where( (np.arange(emb_train.shape[0])%20)==7 )[0]\nlen(train_index), len(valid_index)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Fit each label and predict test using the embeddings features"},{"metadata":{"trusted":true},"cell_type":"code","source":"ytarget = train[targets].values[valid_index]\nypred = np.zeros( (len(valid_index), len(targets)) )\n\nfor n, target in tqdm(enumerate(targets), total=len(targets)):\n    \n    rf = cuml.ensemble.RandomForestClassifier(n_estimators=300, max_features=450, n_bins=16, output_type='numpy')\n    \n    rf.fit( emb_train[train_index], train[target].values[train_index] )\n    \n    ypred[:,n] = rf.predict_proba(emb_train[valid_index])[:,1]\n    test[target] = rf.predict_proba(emb_test)[:,1]\n    \n    print(n, roc_auc_score( ytarget[:,n], ypred[:,n] ), target )\n    \n    del rf\n    gc.collect()\n    \nprint('Final AUC:', roc_auc_score( ytarget.flatten(), ypred.flatten() ) )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Check test predictions distribution"},{"metadata":{"trusted":true},"cell_type":"code","source":"test.mean()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Submit"},{"metadata":{"trusted":true},"cell_type":"code","source":"test.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}