{"cells":[{"metadata":{},"cell_type":"markdown","source":"**Problem:** Classifcation of chest x-rays for malpositioned lines and tubes in patients.\n\n**Evaluation:** Mean AUROC scores."},{"metadata":{},"cell_type":"markdown","source":"## Imports\n\nLets also install `tensorflow_addons` to train using Cyclical Learning Rate.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip3 install tensorflow_addons -q","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# inbuilt imports\nimport os\nimport glob\nimport pathlib\nimport tempfile\nimport functools\n\n# numeric imports\nimport numpy as np\nimport pandas as pd\n\n# visual imports\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\n# modeling imports\nimport tensorflow as tf\nimport tensorflow_addons as tfa","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%matplotlib inline\nsns.set()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load dataset and analyse"},{"metadata":{"trusted":true},"cell_type":"code","source":"base_path = pathlib.Path('..')\npath = pathlib.Path('../input/ranzcr-clip-catheter-line-classification')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Set up constants and hyperparameters\n\nLets us set some values that will remain constant and hyperparameters so that they are easy to modify."},{"metadata":{"trusted":true},"cell_type":"code","source":"IMG_SIZE = 224\nBATCH_SIZE = 16\n\nall_labels = ['ETT - Abnormal', 'ETT - Borderline', 'ETT - Normal', \n              'NGT - Abnormal', 'NGT - Borderline','NGT - Incompletely Imaged', \n              'NGT - Normal', 'CVC - Abnormal', 'CVC - Borderline', 'CVC - Normal', \n              'Swan Ganz Catheter Present']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv(path/'train.csv', low_memory=False)\n\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Hmmm..I can see a PatientID column here. This rings some bells. Lets check the total examples/data points and total unique patient ids.\n\nAnother thing that we can see that a record can have multiple detections. That is, it is multiclass classification problem."},{"metadata":{},"cell_type":"markdown","source":"### Further analysis\n\nLets further check what we stated above is true."},{"metadata":{"trusted":true},"cell_type":"code","source":"len(train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"len(train.PatientID.unique())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"grouped_df = train.set_index('PatientID')\n\ngrouped_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"grouped_df.loc['ec89415d1'].sort_values(by=['StudyInstanceUID'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As suspected, there are multiple records for the same patient. We need to handle this while splitting to prevent any data leaks. If we simply split the data into training and validation set then some of the records of the same patient might end up in validation set and hence the model would have already seen the records of those patient during training. In this case the model can use the already seen information to make predictions during evaluation while validation. This will show good result during validation whereas in reality the model has not learnt anything useful.\n\nSo, instead we will group the data by patient and the split the grouped data instead. In this way we can ensure that all the records of the same patient end up in the same split. Splitting this way is not very optimal as the data is not split stratificaly, but atleast we can address the data leak problem."},{"metadata":{},"cell_type":"markdown","source":"## Data set-up and data pipeline creation\n\nNow, lets add the path of the images in our dataframe so that we can easily utilize the ImageDataGenerator."},{"metadata":{"trusted":true},"cell_type":"code","source":"train['path'] = train['StudyInstanceUID'].map(lambda x: str(path/'train'/(x+'.jpg')))\n\nprint(train.loc[0, ['StudyInstanceUID', 'path']].values)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Analyse the data per classes"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize=(6, 5))\n\ndata = train.loc[:, all_labels].sum().sort_values()\ndist = pd.DataFrame({'Class': data.index, 'Pos': data.values}, \n                    columns=['Class', 'Pos'])\n\nsns.barplot(data=dist, y='Class', x='Pos')\nplt.xticks(rotation=45)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The shows that our data is very imbalanced for each class. We will address this problem later in the notebook."},{"metadata":{"trusted":true},"cell_type":"code","source":"dist['Neg'] = len(train) - dist.Pos\ndist.sort_values(by='Pos')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Splitting Data\n\nRemember prviously we talked about splitting the data by grouping them by patient id. Now's the time to do so. Let's go."},{"metadata":{"trusted":true},"cell_type":"code","source":"grouped_df = train.groupby('PatientID')\n\ntrain_list = [group for _, group in grouped_df]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def train_valid_splitter(d, train_size=0.8):\n    n = len(d)\n    trains = d[:int(train_size*n)]\n    valids = d[int(train_size*n):]\n    return trains, valids\n\ntrain_split, valid_split = train_valid_splitter(train_list)\ntrain_df = pd.concat(train_split, axis=0)\nvalid_df = pd.concat(valid_split, axis=0)\n\nprint(f'Train Size: {len(train_df)}, Valid Size: {len(valid_df)}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Creating data generators\n\nLets use the `ImagDataGenerator` module to build the input pipeline for the model. \n\nAs for augmentations, we are only horizontally flipping and rotating (by 0.4 radians, just a randomly chosen number) images."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_generator = tf.keras.preprocessing.image.ImageDataGenerator(\n    rescale=1./255,\n    horizontal_flip=True,\n    rotation_range=0.40\n)\n\ntrain_datagen = train_generator.flow_from_dataframe(\n    dataframe=train_df,\n    x_col='path',\n    y_col=all_labels,\n    target_size=(IMG_SIZE, IMG_SIZE),\n    batch_size=BATCH_SIZE,\n    seed=42,\n    class_mode='raw'\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"valid_generator = tf.keras.preprocessing.image.ImageDataGenerator(rescale=1./255)\n\nvalid_datagen = valid_generator.flow_from_dataframe(\n    dataframe=valid_df,\n    x_col='path',\n    y_col=all_labels,\n    target_size=(IMG_SIZE, IMG_SIZE),\n    batch_size=BATCH_SIZE,\n    seed=42,\n    class_mode='raw'\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def display_batch(batch, n_imgs=9):\n    r = int(n_imgs**0.5)\n    fig, axs = plt.subplots(r, r, figsize=(12, 15))\n    imgs, labels = batch[0], batch[1]\n    for i, ax in zip(range(n_imgs), axs.flatten()):\n        title = '\\n'.join(list(np.array(all_labels)[labels[i].flatten()==1]))\n        ax.imshow(imgs[i], cmap='bone')\n        ax.set_title(title)\n        ax.grid(False)\n        ax.set_xticks([])\n        ax.set_yticks([])\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"display_batch(next(train_datagen))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Model building\n\nLets start building the model. As a baseline, first I used a pretrained ResNet18. The performance achieved was not very good. So, I moved to the next level, thus using DenseNet121.\n\n**Note: We have not addressed the data imbalance problem yet. The simplest way of dealing with it, will be using weighted binary crossentropy.**\n\n**Note: All the code for training has been commented and the logs been put in markdown to prevent wastage of compute during commit.**"},{"metadata":{"trusted":true},"cell_type":"code","source":"# !pip3 install image-classifiers -q","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# from classification_models.tfkeras import Classifiers\n\n# ResNet18, proc_func = Classifiers.get('resnet18')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As ResNet18 pretrained model is not available in the keras application. We used the one provided by --. But now we have moved to DenseNet121 and hence have commeted the code."},{"metadata":{"trusted":true},"cell_type":"code","source":"densenet = tf.keras.applications.DenseNet121(weights='imagenet', include_top=False, input_shape=(IMG_SIZE, IMG_SIZE, 3))\ndensenet.trainable = False\n\ninputs = densenet.inputs\nx = densenet(inputs)\n\nx = tf.keras.layers.GlobalAveragePooling2D()(x)\noutputs = tf.keras.layers.Dense(len(all_labels), activation='sigmoid')(x)\nmodel = tf.keras.Model(inputs, outputs)\n\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Learning Rate\n\nLets build a learning rate finder to find an optimal lr for finetuning the model."},{"metadata":{"trusted":true},"cell_type":"code","source":"class LRFinder:\n    def __init__(self, model, lr_range=[1e-10, 1e1], beta=0.98, stop_factor=4):\n        self.model = model\n        self.lr_range = lr_range\n        self.beta = beta\n        self.stop_factor = stop_factor\n        self.stop_training = False\n        self.iterations = 0\n        self.mvg_avg_loss = 0\n        self.min_loss = 1e9\n        self.lrs = []\n        self.losses = []\n\n    def _reset(self):\n        self.stop_training = False\n        self.iterations = 0\n        self.mvg_avg_loss = 0\n        self.min_loss = 1e9\n        self.lrs = []\n        self.losses = []\n\n    def _scheduler(self, start_lr, end_lr, iterations):\n        self.lr_factor = (end_lr / start_lr)**(1./iterations)\n\n    def on_train_begin(self, logs=None):\n        self._reset()\n\n    def on_batch_end(self, batch, logs=None):\n        self.iterations += 1\n\n        lr = tf.keras.backend.get_value(self.model.optimizer.lr)\n        self.lrs.append(lr)\n        tf.keras.backend.set_value(self.model.optimizer.lr, lr*self.lr_factor)\n\n        loss = logs['loss']\n        self.mvg_avg_loss = (self.beta*self.mvg_avg_loss) + ((1-self.beta)*loss)\n        smooth_loss = self.mvg_avg_loss / (1-(self.beta**self.iterations))\n        self.losses.append(smooth_loss)\n\n        stop_loss = self.stop_factor * self.min_loss\n        if self.iterations > 1 and smooth_loss > self.stop_factor:\n            self.stop_training = True\n\n        if self.iterations == 0 or smooth_loss < self.min_loss:\n            self.min_loss = smooth_loss\n#         print(f'\\nIterations: {self.iterations}, lr: {lr}, loss: {smooth_loss}/{loss}, lrf: {self.lr_factor}')\n\n    def on_epoch_end(self, epoch, logs=None):\n        if self.stop_training:\n            self.model.stop_training = True\n            return\n\n    def find(self, train_ds, epochs=None, steps_per_epoch=None, batch_size=32):\n        if epochs is None:\n            raise ValueError(f'Invalid value {epochs} for epochs')\n\n        if steps_per_epoch is None:\n            steps_per_epoch = len(train_ds)\n            \n        self._scheduler(self.lr_range[0], self.lr_range[1], steps_per_epoch*epochs)\n\n        with tempfile.NamedTemporaryFile(prefix='init', suffix='.h5') as init_config:\n            # save model config\n            self.model.save_weights(init_config.name)\n            init_lr = tf.keras.backend.get_value(self.model.optimizer.lr)\n            \n            tf.keras.backend.set_value(self.model.optimizer.lr, self.lr_range[0])\n\n            lr_finder_cb = tf.keras.callbacks.LambdaCallback(\n                on_train_begin= lambda logs: self.on_train_begin(logs),\n                on_batch_end= lambda batch, logs: self.on_batch_end(batch, logs),\n                on_epoch_end= lambda epoch, logs: self.on_epoch_end(epoch, logs)\n            )\n\n            self.model.fit(train_ds, epochs=epochs, steps_per_epoch=steps_per_epoch,\n                           callbacks=[lr_finder_cb])\n\n            # restore model config\n            tf.keras.backend.set_value(self.model.optimizer.lr, init_lr)\n            self.model.load_weights(init_config.name)\n\n    def plot_loss(self, skip_begin=10, skip_end=1, title=\"\"):\n        lrs = self.lrs[skip_begin:-skip_end]\n        losses = self.losses[skip_begin:-skip_end]\n        plt.plot(lrs, losses)\n        plt.xscale(\"log\")\n        plt.xlabel(\"Learning Rate (Log Scale)\")\n        plt.ylabel(\"Loss\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# model.compile(optimizer='adam',\n#               loss='binary_crossentropy',\n#               metrics=['binary_accuracy', tf.keras.metrics.AUC()])\n\n# lr_finder = LRFinder(model)\n# lr_finder.find(train_datagen, epochs=10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# lr_finder.plot_loss()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"![](https://storage.googleapis.com/kagglesdsdata/datasets/1225983/2068384/lrfinder.png?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=databundle-worker-v2%40kaggle-161607.iam.gserviceaccount.com%2F20210329%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20210329T145149Z&X-Goog-Expires=172799&X-Goog-SignedHeaders=host&X-Goog-Signature=3e445be1f2c3d9455e46b2e540cdb7db59199d602bbe54a7ef185d164b4dc86ae1beaafd9d3817bba456bba35ea0ae45e811653470a36d36018bd896fb299099e18d75bd1349a63097f816fe6ca89832bf2c47e13c8d1d039fa9fa6eea07aef69fc16dedc35f22c2e80c502e53231eddc0a3503a50b22c112524027de61c51eebb9a2c5ecc3255340d47a97f20ccaee668d571dcc96a094ffc4f4a0462f467a316465831e0091058773bb11e661e64a303d3426e4010b13a432eb3b70333d6f161a88db4cc22c921edd4dd46136400b8e2b4de878bbb36d02c280643d8e188c185ae0ff34e1c06f4ab639284cc489cae3b28b8a56e4647e0d450c123d49161b4)"},{"metadata":{},"cell_type":"markdown","source":"For our CLR we can use a minimum learning rate of 1e-2 and a maximum learning rate of 1e-5."},{"metadata":{},"cell_type":"markdown","source":"Lets use the CyclicalLearningRate provided by `tensorflow_addons` library, keeping the initial learning rate to 1e-5 and the maximum learning rate to be 1e-2 and use a 'traingular' approach of CLR. We are keeping the step size = 2 epochs. Hence, to perform 1 cycle we will need to perform 4 epochs."},{"metadata":{"trusted":true},"cell_type":"code","source":"def scale_fn(x):  return 1.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# clr = tfa.optimizers.CyclicalLearningRate(\n#     initial_learning_rate=1e-5,\n#     maximal_learning_rate=1e-2,\n#     scale_fn=scale_fn,\n#     step_size=2*len(train_datagen)*BATCH_SIZE,\n#     scale_mode='cyclic'\n# )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"To evaluate our model we will use 'binary_accuracy' and 'AUROC Score', as our model itself will be evaluated on the AUROC score."},{"metadata":{"trusted":true},"cell_type":"code","source":"# model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=clr),\n#               loss='binary_crossentropy',\n#               metrics=['binary_accuracy', tf.keras.metrics.AUC()])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# model.fit(train_datagen, epochs=4, batch_size=BATCH_SIZE, \n#           steps_per_epoch=len(train_datagen), \n#           validation_data=valid_datagen,\n#           validation_steps=len(valid_datagen))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Logs for fine-tuning for 1 cycle (Frozen pre-trained model)\n\nEpoch 1/4\n1507/1507 [==============================] - 1215s 801ms/step - loss: 0.4272 - binary_accuracy: 0.7996 - auc_2: 0.7669 - val_loss: 0.2668 - val_binary_accuracy: 0.8886 - val_auc_2: 0.8993\n\nEpoch 2/4\n1507/1507 [==============================] - 1221s 811ms/step - loss: 0.2634 - binary_accuracy: 0.8909 - auc_2: 0.9037 - val_loss: 0.2560 - val_binary_accuracy: 0.8892 - val_auc_2: 0.9080\n\nEpoch 3/4\n1507/1507 [==============================] - 1217s 807ms/step - loss: 0.2555 - binary_accuracy: 0.8929 - auc_2: 0.9105 - val_loss: 0.2560 - val_binary_accuracy: 0.8916 - val_auc_2: 0.9098\n\nEpoch 4/4\n1507/1507 [==============================] - 1217s 808ms/step - loss: 0.2507 - binary_accuracy: 0.8938 - auc_2: 0.9145 - val_loss: 0.2523 - val_binary_accuracy: 0.8927 - val_auc_2: 0.9134\n"},{"metadata":{},"cell_type":"markdown","source":"Now lets unfreeze our model and finetune the entire model."},{"metadata":{"trusted":true},"cell_type":"code","source":"densenet.trainable = True\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# clr = tfa.optimizers.CyclicalLearningRate(\n#     initial_learning_rate=1e-9,\n#     maximal_learning_rate=1e-4,\n#     scale_fn=scale_fn,\n#     step_size=2*len(train_datagen)*BATCH_SIZE,\n#     scale_mode='cyclic'\n# )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=clr),\n#               loss='binary_crossentropy',\n#               metrics=['binary_accuracy', tf.keras.metrics.AUC()])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# model.fit(train_datagen, epochs=4, batch_size=BATCH_SIZE, \n#           steps_per_epoch=len(train_datagen), \n#           validation_data=valid_datagen,\n#           validation_steps=len(valid_datagen))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Logs for fine-tuning for 1 cycle (Unfrozen pre-trained model)\n\nEpoch 1/4\n1507/1507 [==============================] - 1338s 878ms/step - loss: 0.3342 - binary_accuracy: 0.8664 - auc_3: 0.8492 - val_loss: 0.2725 - val_binary_accuracy: 0.8832 - val_auc_3: 0.8958\n\nEpoch 2/4\n1507/1507 [==============================] - 1297s 861ms/step - loss: 0.2648 - binary_accuracy: 0.8899 - auc_3: 0.9045 - val_loss: 0.2523 - val_binary_accuracy: 0.8907 - val_auc_3: 0.9123\n\nEpoch 3/4\n1507/1507 [==============================] - 1291s 857ms/step - loss: 0.2425 - binary_accuracy: 0.8967 - auc_3: 0.9216 - val_loss: 0.2401 - val_binary_accuracy: 0.8957 - val_auc_3: 0.9219\n\nEpoch 4/4\n1507/1507 [==============================] - 1283s 851ms/step - loss: 0.2243 - binary_accuracy: 0.9048 - auc_3: 0.9343 - val_loss: 0.2319 - val_binary_accuracy: 0.8941 - val_auc_3: 0.9277"},{"metadata":{"trusted":true},"cell_type":"code","source":"# model.save('finetuned.h5')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model = tf.keras.models.load_model(base_path/'input'/'extras'/'ranzcr-pretrained.h5',\n                                   custom_objects={'scale_fn': scale_fn})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_files = {\n    os.path.basename(f)[:-4] : f\n    for f in glob.glob(str(path/'test'/'*.jpg'))\n}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df = pd.DataFrame({'StudyInstanceUID': list(test_files.keys()), 'path': list(test_files.values())})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def test_predictor(df, model):\n    res = {\n        l: []\n        for l in all_labels\n    }\n    for f, p in df.values:\n        img = tf.keras.preprocessing.image.load_img(p, target_size=(IMG_SIZE, IMG_SIZE))\n        img = np.expand_dims(np.asarray(img)/225, axis=0)\n        pred = model.predict(img).flatten()\n        for i in range(len(all_labels)):\n            res[all_labels[i]].append(pred[i])\n    return res\n\nres = test_predictor(test_df, model)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"r = pd.DataFrame(res)\nsubmission = pd.concat([test_df['StudyInstanceUID'], r], axis=1)\nsubmission.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.evaluate(valid_datagen)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}