{"cells":[{"metadata":{},"cell_type":"markdown","source":"# π Overview\n* In this notebook,we build best model using xception with some refrences\n* Using open cv base augmentation \n* This baseline notebook using TPU is a part of next final notebook prediction using GPU will coming soon!"},{"metadata":{},"cell_type":"markdown","source":"## π [Part 2] -> [RANZCR]: Detection With Xception | Final Session"},{"metadata":{},"cell_type":"markdown","source":"# π Getting familiar with the dataset\nThe dataset contains a lot of biological terms which may be confusing if you haven't looked back at your biology lessons for a long time. But we will discuss them together.\n* Endotracheal tube(ETT) - This is nothing but a tube passed through the trachea(or windpipe) which is a pipe like structure which helps in the passage of air(breathing). Endotracheal tube is a flexible plastic tube placed through the mouth into the trachea to help a patient breathe.\n* Nasogastric tube(NGT) - This is a flexible tube made of either rubber or plastic which is passed through the nose to the esophagus(a tube connecting the throat with the stomach) into the stomach.This is used for feeding food and medicine to the stomach through the nose.\n* Central venous catheter(CVC) - This is a thin, flexible tube inserted into the body through a vein usually below the collar bone and is guided into a large vein above the right side of the heart called superior vena cava. CVC is introduced to give fluids, blood or other drugs."},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nimport math\nimport pandas as pd\nimport seaborn as sns\nimport albumentations as A \nimport matplotlib.pyplot as plt\nimport plotly.graph_objects as go\nimport os, gc, cv2, random, warnings, math, sys, json, pprint\nfrom glob import glob\nfrom pylab import rcParams\n\n# sklearn\nfrom sklearn.model_selection import GroupKFold\nfrom sklearn.metrics import roc_auc_score\n\n# tf \nimport tensorflow as tf\n#import efficientnet.tfkeras as efn\nfrom tensorflow.keras import backend as K\n\n# torch\nimport torch\nimport torchvision\nfrom torchvision import transforms\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data_dir = '../input/ranzcr-clip-catheter-line-classification'\nos.listdir(data_dir)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Train images: %d' %len(os.listdir(os.path.join(data_dir, \"train\"))))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from kaggle_datasets import KaggleDatasets\n\ndata_dir ='../input/ranzcr-clip-catheter-line-classification'\npath_dir = KaggleDatasets().get_gcs_path('ranzcr-clip-catheter-line-classification')\n\ntrain = os.path.join(data_dir, 'train.csv')\ntrain_df = pd.read_csv(train) \nprint(\"data_train_csv : \" + str(train_df.shape[0]))\ndisplay(train_df.head(5))\n\n\nsub = os.path.join(data_dir,'sample_submission.csv')\nsub_df = pd.read_csv(sub)\nprint(\"data_submission_csv : \" + str(sub_df.shape[0]) )\ndisplay(sub_df.head(5))\n\nlabel_cols = sub_df.columns[1:]\n#label_cols.values\nlabels = train_df[label_cols].values\n\ntrain_images = path_dir + \"/train/\" + train_df['StudyInstanceUID'] + '.jpg' \ntest_images = path_dir + \"/test/\" + sub_df['StudyInstanceUID'] + '.jpg'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_image = 0\ntest_image = 0\n\ndir1 = '../input/ranzcr-clip-catheter-line-classification/train'\ndir2 = '../input/ranzcr-clip-catheter-line-classification/test'\n\n\ndef data_image(dir1,dir2,train_image,test_image):\n dir_train = len(dir1)\n train_image += dir_train\n if dir_train:\n print(f\"Train Image : {train_image}\")\n dir_test = len(dir2)\n test_image += dir_test\n if dir_test: \n print(f\"Test Image : {test_image}\")\n \ndata_image(dir1,dir2,train_image,test_image)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data_image = os.path.join(data_dir, \"train_annotations.csv\")\ndata_image = pd.read_csv(data_image)\ndata_image.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# πΈ Display All Image"},{"metadata":{"trusted":true},"cell_type":"code","source":"def all_image(image_ids,label_ids, binarizer):\n fig = plt.figure(figsize = (16,12))\n \n '''\n Display all image with of various kinds of labels\n '''\n \n for ind, (image_id, label) in enumerate(zip(image_ids, label_ids)):\n plt.subplot(4, 4, ind+1)\n image = image_id + '.jpg'\n image = cv2.imread(os.path.join(data_dir, \"train\", image))\n image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n rnd_number = randint(0,len(image_id))\n bin_class = binarizer[ind]\n\n plt.imshow(image)\n title = f'{label}\\n{bin_class}'\n plt.title(title, fontsize = 12)\n #plt.title(f\"Class: {label}\\nf\"Binary: {bin_class}\",fontsize=12)\n plt.axis(\"on\")\n \n fig.tight_layout()\n plt.show() ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.preprocessing import LabelBinarizer\nfrom random import randint\n\ntmp_train = data_image.sample(16)\nimage_ids = tmp_train[\"StudyInstanceUID\"].values\nlabel_ids = tmp_train[\"label\"].values\n\nlb_category = LabelBinarizer()\nbinarizer = lb_category.fit_transform(label_ids)\n\nall_image(image_ids, label_ids, binarizer)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# π Display Data Plot Images"},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_categorical(train, label_cols):\n fig = plt.figure(figsize = (15, 13), dpi = 300)\n plt.suptitle('Labels count', fontfamily = 'serif', size = 15)\n\n \n for ind, i in enumerate(label_cols):\n fig.add_subplot(4, 3, ind + 1)\n sns.countplot(data = train_df,\n x = i,\n #train[i], \n edgecolor = 'black',\n palette = reversed(sns.color_palette('mako', 2)))\n \n plt.xlabel('')\n plt.ylabel('')\n plt.xticks(fontfamily = 'serif', size = 10)\n plt.yticks(fontfamily = 'serif', size = 10)\n plt.title(i, fontfamily = 'serif', size = 10)\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_categorical(train,label_cols)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"'''\n TPU or GPU detection\n Detect hardware, return appropriate distribution strategy\n'''\ntry:\n tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n print(f'Running on TPU {tpu.master()}')\nexcept ValueError:\n tpu = None\n\nif tpu:\n tf.config.experimental_connect_to_cluster(tpu)\n tf.tpu.experimental.initialize_tpu_system(tpu)\n strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n strategy = tf.distribute.get_strategy()\n\nAUTO = tf.data.experimental.AUTOTUNE\nREPLICAS = strategy.num_replicas_in_sync\nprint(f'REPLICAS: {REPLICAS}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# π Open CV - Augmentation\n**What is Data Augmentation?**\n* Moving, rotating, cropping, flipping, changing color/brightness/hue and whatever else you can come up with to change the aspect of the original image. It is helpful in Overfitting, as the model learns not only 1 aspect of the image, but multiple (a cat can be standing up straight, or funny upside down, in a b&w image etc.)\n\n**Other things to keep in mind**\n* Different skin tones. Might need to find something that levels that.\n\n* Different lightings in the image.\n\n* Different sizes of the images. We need to resize them."},{"metadata":{"trusted":true},"cell_type":"code","source":"def NeedleAugmentation(image, n_needles=2, dark_needles=False, p=0.5, needle_folder='../input/xray-needle-augmentation'):\n \n '''\n Open CV - Based Custom Augmentation\n '''\n \n aug_prob = random.random()\n if aug_prob < p:\n height, width, _ = image.shape # target image width and height\n needle_images = [im for im in os.listdir(needle_folder) if 'png' in im]\n\n for _ in range(1, n_needles):\n needle = cv2.cvtColor(cv2.imread(os.path.join(needle_folder, random.choice(needle_images))), cv2.COLOR_BGR2RGB)\n needle = cv2.flip(needle, random.choice([-1, 0, 1]))\n needle = cv2.rotate(needle, random.choice([0, 1, 2]))\n\n h_height, h_width, _ = needle.shape # needle image width and height\n roi_ho = random.randint(0, abs(image.shape[0] - needle.shape[0]))\n roi_wo = random.randint(0, abs(image.shape[1] - needle.shape[1]))\n roi = image[roi_ho:roi_ho + h_height, roi_wo:roi_wo + h_width]\n\n # Creating a mask and inverse mask \n img2gray = cv2.cvtColor(needle, cv2.COLOR_BGR2GRAY)\n ret, mask = cv2.threshold(img2gray, 10, 255, cv2.THRESH_BINARY)\n mask_inv = cv2.bitwise_not(mask)\n\n # Now black-out the area of needle in ROI\n img_bg = cv2.bitwise_and(roi, roi, mask=mask_inv)\n\n # Take only region of insect from insect image.\n if dark_needles:\n img_bg = cv2.bitwise_and(roi, roi, mask=mask_inv)\n needle_fg = cv2.bitwise_and(img_bg, img_bg, mask=mask)\n else:\n needle_fg = cv2.bitwise_and(needle, needle, mask=mask)\n\n # Put needle in ROI and modify the target image\n dst = cv2.add(img_bg, needle_fg, dtype=cv2.CV_64F)\n\n image[roi_ho:roi_ho + h_height, roi_wo:roi_wo + h_width] = dst\n\n return image","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Main parameters\nBATCH_SIZE = 8 * REPLICAS\nSTEPS_PER_EPOCH = len(train_df) * 0.8 / BATCH_SIZE\nVALIDATION_STEPS = len(train_df) * 0.2 / BATCH_SIZE\nEPOCHS = 30\nTARGET_SIZE = 750","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_decoder(with_labels = True,\n target_size = (TARGET_SIZE, TARGET_SIZE), \n ext = 'jpg'):\n def decode(path):\n file_bytes = tf.io.read_file(path)\n if ext == 'png':\n img = tf.image.decode_png(file_bytes, channels = 3)\n elif ext in ['jpg', 'jpeg']:\n img = tf.image.decode_jpeg(file_bytes, channels = 3)\n else:\n raise ValueError(\"Image extension not supported\")\n\n img = tf.cast(img, tf.float32) / 255.0\n img = tf.image.resize(img, target_size)\n\n return img\n \n def decode_with_labels(path, label):\n return decode(path), label\n \n return decode_with_labels if with_labels else decode\n\n\ndef build_augmenter(with_labels = True):\n def augment(img):\n img = NeedleAugmentation(img, n_needles=2, dark_needles=False, p=0.5)\n img = tf.image.random_flip_left_right(img)\n img = tf.image.random_flip_up_down(img)\n return img\n \n def augment_with_labels(img, label):\n return augment(img), label\n \n return augment_with_labels if with_labels else augment\n\n\ndef build_dataset(paths, labels = None, bsize = 32, cache = True,\n decode_fn = None, augment_fn = None,\n augment = True, repeat = True, shuffle = 1024, \n cache_dir = \"\"):\n if cache_dir != \"\" and cache is True:\n os.makedirs(cache_dir, exist_ok=True)\n \n if decode_fn is None:\n decode_fn = build_decoder(labels is not None)\n \n if augment_fn is None:\n augment_fn = build_augmenter(labels is not None)\n \n AUTO = tf.data.experimental.AUTOTUNE\n slices = paths if labels is None else (paths, labels)\n \n dset = tf.data.Dataset.from_tensor_slices(slices)\n dset = dset.map(decode_fn, num_parallel_calls = AUTO)\n dset = dset.cache(cache_dir) if cache else dset\n dset = dset.map(augment_fn, num_parallel_calls = AUTO) if augment else dset\n dset = dset.repeat() if repeat else dset\n dset = dset.shuffle(shuffle) if shuffle else dset\n dset = dset.batch(bsize).prefetch(AUTO)\n \n return dset","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nimport tensorflow as tf\nfrom tensorflow.keras import models, layers\nfrom tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping, ReduceLROnPlateau\nfrom tensorflow.keras.applications import Xception\nfrom tensorflow.keras.optimizers import Adam\n\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\n# ignoring warnings\nimport warnings\nwarnings.simplefilter(\"ignore\")\n\nimport os, cv2\n\n# Train test split\n(train_img, valid_img, train_labels, valid_labels) = train_test_split(train_images, labels, train_size = 0.8, random_state = 0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Tensorflow datasets\ntrain_df = build_dataset(\n train_img, train_labels, bsize = BATCH_SIZE, \n cache = True)\n\nvalid_df = build_dataset(\n valid_img, valid_labels, bsize = BATCH_SIZE, \n repeat = False, shuffle = False, augment = False, \n cache = True)\n\ntest_df = build_dataset(\n test_images, bsize = BATCH_SIZE, repeat = False, \n shuffle = False, augment = False, cache = False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# π§ͺ Model Xception\n![](\"https://camo.githubusercontent.com/8fbdfc47b1575c45f4d2132b96f34826f2d913af/68747470733a2f2f6d69726f2e6d656469756d2e636f6d2f6d61782f313430302f312a684f6341456a39517a716742586377557a6d457653672e706e67\")
"},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_model():\n conv_base = Xception(include_top = False, weights = 'imagenet',\n input_shape = (TARGET_SIZE, TARGET_SIZE, 3))\n model = conv_base.output\n model = layers.GlobalAveragePooling2D()(model)\n model = layers.Dropout(0.25)(model)\n model = layers.Dense(11, activation = \"sigmoid\")(model)\n model = models.Model(conv_base.input, model)\n\n model.compile(optimizer = Adam(lr = 0.001),\n loss = \"binary_crossentropy\",\n metrics = [tf.keras.metrics.AUC(multi_label = True)])\n return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n model = create_model()\n \nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Our Xception CNN has %d layers' %len(model.layers))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model_save = ModelCheckpoint('./Xcep_750_best_weights_TPU.h5', \n save_best_only = True, \n save_weights_only = True,\n monitor = 'val_loss', \n mode = 'min', verbose = 1)\n\nearly_stop = EarlyStopping(monitor = 'val_loss', min_delta = 0.0001, \n patience = 5, mode = 'min', verbose = 1,\n restore_best_weights = True)\n\nreduce_lr = ReduceLROnPlateau(monitor = 'val_loss', factor = 0.4, \n patience = 2, min_delta = 0.0001, \n mode = 'min', verbose = 1)\n\n\nhistory = model.fit(\n train_df,\n epochs = EPOCHS,\n steps_per_epoch = STEPS_PER_EPOCH,\n validation_data = valid_df,\n validation_steps = VALIDATION_STEPS,\n callbacks = [model_save, early_stop, reduce_lr]\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"auc = history.history['auc']\nval_auc = history.history['val_auc']\nloss = history.history['loss']\nval_loss = history.history['val_loss']\n\nepochs = range(1, len(auc) + 1)\n\nfig, (ax1, ax2) = plt.subplots(1, 2, figsize = (15, 5))\nsns.set_style(\"white\")\nplt.suptitle('Train history', size = 15)\n\nax1.plot(epochs, auc, \"bo\", label = \"Training auc\")\nax1.plot(epochs, val_auc, \"b\", label = \"Validation auc\")\nax1.set_title(\"Training and validation auc\")\nax1.legend()\n\nax2.plot(epochs, loss, \"bo\", label = \"Training loss\", color = 'red')\nax2.plot(epochs, val_loss, \"b\", label = \"Validation loss\", color = 'red')\nax2.set_title(\"Training and validation loss\")\nax2.legend()\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.save('./Xception_750_TPU.h5')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# πActivations Layer Maps\n**So, what are activation anyway ?**\n* At a simple level, activation functions help decide whether a neuron should be activated. This helps determine whether the information that the neuron is receiving is relevant for the input. The activation function is a non-linear transformation that happens over an input signal, and the transformed output is sent to the next neuron.\n* If you want to understand what precisely, these activations mean, and why are they placed in the neural net architecture in the first place, check out this article below this: \n* [](http://)[https://www.analyticsvidhya.com/blog/2020/01/fundamentals-deep-learning-activation-functions-when-to-use-them/](http://https://www.analyticsvidhya.com/blog/2020/01/fundamentals-deep-learning-activation-functions-when-to-use-them/)"},{"metadata":{"trusted":true},"cell_type":"code","source":"def activation_layer_vis(img, activation_layer = 0, layers = 10):\n layer_outputs = [layer.output for layer in model.layers[:layers]]\n activation_model = models.Model(inputs = model.input, outputs = layer_outputs)\n activations = activation_model.predict(img)\n \n rows = int(activations[activation_layer].shape[3] / 3)\n cols = int(activations[activation_layer].shape[3] / rows)\n fig, axes = plt.subplots(rows, cols, figsize = (15, 15 * cols))\n axes = axes.flatten()\n \n for i, ax in zip(range(activations[activation_layer].shape[3]), axes):\n ax.matshow(activations[activation_layer][0, :, :, i], cmap = 'viridis')\n ax.axis('off')\n plt.tight_layout()\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"img_tensor = build_dataset(\n pd.Series(test_images[0]), bsize = 1,repeat = False, \n shuffle = False, augment = False, cache = False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"activation_layer_vis(img_tensor)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def all_activations_vis(img, layers = 10):\n layer_outputs = [layer.output for layer in model.layers[:layers]]\n activation_model = models.Model(inputs = model.input, outputs = layer_outputs)\n activations = activation_model.predict(img)\n \n layer_names = []\n for layer in model.layers[:layers]: \n layer_names.append(layer.name) \n\n images_per_row = 3\n for layer_name, layer_activation in zip(layer_names, activations): \n n_features = layer_activation.shape[-1] \n\n size = layer_activation.shape[1] \n\n n_cols = n_features // images_per_row \n display_grid = np.zeros((size * n_cols, images_per_row * size)) \n\n for col in range(n_cols): \n for row in range(images_per_row): \n channel_image = layer_activation[0, :, :, col * images_per_row + row] \n channel_image -= channel_image.mean() \n channel_image /= channel_image.std() \n channel_image *= 64 \n channel_image += 128 \n channel_image = np.clip(channel_image, 0, 255).astype('uint8') \n display_grid[col * size : (col + 1) * size, \n row * size : (row + 1) * size] = channel_image \n scale = 1. / size \n plt.figure(figsize=(scale * 5 * display_grid.shape[1], \n scale * 5 * display_grid.shape[0])) \n plt.title(layer_name) \n plt.grid(False)\n plt.axis('off')\n plt.imshow(display_grid, aspect = 'auto', cmap = 'viridis')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"all_activations_vis(img_tensor, 3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**So here it is! Letβs try to interpret whatβs going on:**\n\nReference : [https://towardsdatascience.com/visualizing-intermediate-activation-in-convolutional-neural-networks-with-keras-260b36d60d0](http://https://towardsdatascience.com/visualizing-intermediate-activation-in-convolutional-neural-networks-with-keras-260b36d60d0)\n\n* The first layer is arguably retaining the full shape of the triangle, although there are several filters that are not activated and are left blank. At that stage, the activations retain almost all of the information present in the initial picture.\n\n* As we go deeper in the layers, the activations become increasingly abstract and less visually interpretable. They begin to encode higher-level concepts such as single borders, corners and angles. Higher presentations carry increasingly less information about the visual contents of the image, and increasingly more information related to the class of the image.\n\n* As mentioned above, the model stucture is overly complex to the point where we can see our last layers actually not activating at all, thereβs nothing more to learn at that point."},{"metadata":{"trusted":true},"cell_type":"code","source":"sub_df[label_cols] = model.predict(test_df, verbose = 1)\nsub_df.to_csv('submission.csv', index = False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## References :"},{"metadata":{},"cell_type":"markdown","source":"Some references to notebooks were made in this notebook\n* [https://www.kaggle.com/maksymshkliarevskyi/ranzcr-xception-tpu-prediction](http://www.kaggle.com/maksymshkliarevskyi/ranzcr-xception-tpu-prediction)\n* [https://www.kaggle.com/khoongweihao/insect-augmentation-with-efficientdet-d6](http://www.kaggle.com/khoongweihao/insect-augmentation-with-efficientdet-d6)\n* https://www.kaggle.com/andradaolteanu/siim-melanoma-competition-eda-augmentations\n* https://arxiv.org/abs/1610.02357v3\n"},{"metadata":{},"cell_type":"markdown","source":"# If it's useful for you, come on upvote.Thank you for your attention π"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}