{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# š§ Introduction\n\nHey all, this competition is going to be an interesting one given the nature of the dataset. It's not a straighforward image classification problem and one can forulate the problem statement in multiple ways.\n\nIn my [EDA kernel](http://wandb.me/brain-eda), I have shown that the MRI images per patient and scan type is sequential in nature. \n\nAs a doctor I would be interested to look at multiple slices (images) of the MRI Sequence to determine if a patient has brain tumor or not. Thus I have formulated the brain tumor classification problem statement as Video Classification. \n\nš This is a work in progress and I would love to hear your suggestions to improve the training pipeline. \n\nI am using the dataset created by [Jonathan Besomi](https://www.kaggle.com/jonathanbesomi). Many thanks to him for creating this. You can find the data [here](https://www.kaggle.com/jonathanbesomi/rsna-miccai-png). \n\nI have implemented the pipeline in TensorFlow and using [Weights and Biases](https://wandb.ai/site) for experiment tracking and data visualization. ","metadata":{}},{"cell_type":"markdown","source":"# š Imports and Setup","metadata":{}},{"cell_type":"code","source":"import os\nimport re\nimport gc\nimport glob\nimport imageio\nimport numpy as np\nimport pandas as pd\nfrom tqdm import tqdm\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\nimport tensorflow as tf\nprint('TF version: ', tf.__version__)\nfrom tensorflow.keras.layers import *\nfrom tensorflow.keras.models import *\n\nfrom sklearn.model_selection import train_test_split","metadata":{"execution":{"iopub.status.busy":"2021-07-21T21:50:16.275387Z","iopub.execute_input":"2021-07-21T21:50:16.275782Z","iopub.status.idle":"2021-07-21T21:50:21.275281Z","shell.execute_reply.started":"2021-07-21T21:50:16.275698Z","shell.execute_reply":"2021-07-21T21:50:21.274479Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Set up Weights and Biases","metadata":{}},{"cell_type":"code","source":"import wandb\nprint('W&B version: ', wandb.__version__)\nfrom wandb.keras import WandbCallback\n\nwandb.login()","metadata":{"execution":{"iopub.status.busy":"2021-07-21T21:50:25.938193Z","iopub.execute_input":"2021-07-21T21:50:25.938528Z","iopub.status.idle":"2021-07-21T21:50:35.132182Z","shell.execute_reply.started":"2021-07-21T21:50:25.938496Z","shell.execute_reply":"2021-07-21T21:50:35.131316Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gpus = tf.config.list_physical_devices('GPU')\nif gpus:\n try:\n # Currently, memory growth needs to be the same across GPUs\n for gpu in gpus:\n tf.config.experimental.set_memory_growth(gpu, True)\n logical_gpus = tf.config.experimental.list_logical_devices('GPU')\n print(len(gpus), \"Physical GPUs,\", len(logical_gpus), \"Logical GPUs\")\n except RuntimeError as e:\n # Memory growth must be set before GPUs have been initialized\n print(e)","metadata":{"execution":{"iopub.status.busy":"2021-07-21T21:50:38.334907Z","iopub.execute_input":"2021-07-21T21:50:38.335229Z","iopub.status.idle":"2021-07-21T21:50:40.066397Z","shell.execute_reply.started":"2021-07-21T21:50:38.335196Z","shell.execute_reply":"2021-07-21T21:50:40.065432Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# š Prepare Dataset\n\nThere are four sub-directories per patient corresponding to different MRI Image Sequencing methods. In this kernel, I am using \"FLAIR\" MRI to get the balls rolling. To get the maximum out of the dataset using every sequencing method is recommended. ","metadata":{}},{"cell_type":"code","source":"# Load training csv file\ndf = pd.read_csv('../input/rsna-miccai-brain-tumor-radiogenomic-classification/train_labels.csv')\n\ndef get_patient_id(patient_id):\n if patient_id < 10:\n return '0000'+str(patient_id)\n elif patient_id >= 10 and patient_id < 100:\n return '000'+str(patient_id)\n elif patient_id >= 100 and patient_id < 1000:\n return '00'+str(patient_id)\n else:\n return '0'+str(patient_id)\n\ndef get_path(row):\n patient_id = get_patient_id(row.BraTS21ID)\n return f'../input/rsna-miccai-png/train/{patient_id}/FLAIR/'\n\ndf['path'] = df.apply(lambda row: get_path(row), axis=1)\n\n# Removing two patient ids from the dataframe since there are not FLAIR directories for these ids. \ndf = df.loc[df.BraTS21ID!=109]\ndf = df.loc[df.BraTS21ID!=709]\ndf = df.reset_index(drop=True)\n\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-21T22:06:14.236894Z","iopub.execute_input":"2021-07-21T22:06:14.237224Z","iopub.status.idle":"2021-07-21T22:06:14.275307Z","shell.execute_reply.started":"2021-07-21T22:06:14.237192Z","shell.execute_reply":"2021-07-21T22:06:14.274447Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Prepare train-test split. Note that there are only 585 patients so if you are doing video classification, K-fold training might be beneficial. ","metadata":{}},{"cell_type":"code","source":"train_df, valid_df = train_test_split(df, test_size=0.1, stratify=df.MGMT_value.values)\nprint(f'Size of train_df: {len(train_df)}; valid_df: {len(valid_df)}')","metadata":{"execution":{"iopub.status.busy":"2021-07-21T22:06:17.327404Z","iopub.execute_input":"2021-07-21T22:06:17.327808Z","iopub.status.idle":"2021-07-21T22:06:17.335582Z","shell.execute_reply.started":"2021-07-21T22:06:17.327776Z","shell.execute_reply":"2021-07-21T22:06:17.334549Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"CONFIG = dict(\n NUM_FRAMES = 10,\n BATCH_SIZE = 8,\n EPOCHS = 100,\n IMG_SIZE = 224,\n LSTM_UNITS = 256,\n competition = 'rsna-miccai-brain',\n _wandb_kernel = 'ayut'\n)","metadata":{"execution":{"iopub.status.busy":"2021-07-21T22:06:18.927266Z","iopub.execute_input":"2021-07-21T22:06:18.927651Z","iopub.status.idle":"2021-07-21T22:06:18.932349Z","shell.execute_reply.started":"2021-07-21T22:06:18.927615Z","shell.execute_reply":"2021-07-21T22:06:18.931391Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# š Video Classification Data Pipeline\n\nA video classification data pipeline will compromise of multiple frames of the same video batched together. In order to batch the frames, the number of frames should be same. In this kerel it's controlled by `NUM_FRAMES`. \n\nIn have implemented the data pipeline using purely `tf.data`. Here are the important points to note:\n\n* The images in each `patient_id/FLAIR` directory is listed down using `glob.glob`.
\n* The path to images need to be sorted as per the image id given by `Image-X.png`. This is done by `sorted_nicely` function below.
\n* We need to select a window of frames given by `NUM_FRAMES`. I am using uniform sampling to do so. One can device a better sampling method.
\n* Iterate through each frame (image), load them and resize them. š ","metadata":{}},{"cell_type":"code","source":"# https://stackoverflow.com/a/2669120/7636462\ndef sorted_nicely(l): \n \"\"\" Sort the given iterable in the way that humans expect.\"\"\" \n convert = lambda text: int(text) if text.isdigit() else text \n alphanum_key = lambda key: [ convert(c) for c in re.split('([0-9]+)', key) ] \n return sorted(l, key = alphanum_key)","metadata":{"execution":{"iopub.status.busy":"2021-07-21T22:06:22.038134Z","iopub.execute_input":"2021-07-21T22:06:22.038448Z","iopub.status.idle":"2021-07-21T22:06:22.044891Z","shell.execute_reply.started":"2021-07-21T22:06:22.038418Z","shell.execute_reply":"2021-07-21T22:06:22.042424Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def decode_image(image):\n # convert the compressed string to a 3D uint8 tensor\n image = tf.image.decode_png(image, channels=1)\n # Normalize image\n image = tf.image.convert_image_dtype(image, dtype=tf.float32)\n \n return image\n\ndef parse_frames(dirname):\n # get MRI images file paths for given patient \n paths = glob.glob(dirname.decode('utf8')+'/*.png')\n # Sort the images to get sequential imaging\n paths = sorted_nicely(paths)\n \n # randomly select a window of images to be used as sequence\n start = tf.random.uniform((1,), maxval=len(paths)-CONFIG['NUM_FRAMES'], dtype=tf.int32)\n\n paths = tf.slice(paths, start, [CONFIG['NUM_FRAMES']])\n \n def get_frames(path):\n # Load image\n image = tf.io.read_file(path)\n image = decode_image(image)\n # Resize image\n image = tf.image.resize(image, (CONFIG['IMG_SIZE'], CONFIG['IMG_SIZE']))\n \n return image\n\n mri_images = tf.nest.map_structure(tf.stop_gradient, tf.map_fn(fn=get_frames, elems=paths, fn_output_signature=tf.float32))\n \n return mri_images\n \ndef load_frame(df_dict):\n dirname = df_dict['path']\n paths = tf.numpy_function(parse_frames, [dirname], tf.float32)\n \n # Parse label\n label = df_dict['MGMT_value']\n label = tf.cast(label, tf.float32)\n \n return paths, label","metadata":{"execution":{"iopub.status.busy":"2021-07-21T22:06:33.402212Z","iopub.execute_input":"2021-07-21T22:06:33.402534Z","iopub.status.idle":"2021-07-21T22:06:33.411734Z","shell.execute_reply.started":"2021-07-21T22:06:33.402495Z","shell.execute_reply":"2021-07-21T22:06:33.410829Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"AUTOTUNE = tf.data.AUTOTUNE\n\ntrainloader = tf.data.Dataset.from_tensor_slices(dict(train_df))\nvalidloader = tf.data.Dataset.from_tensor_slices(dict(valid_df))\n\n\ntrainloader = (\n trainloader\n .shuffle(1024)\n .map(load_frame, num_parallel_calls=AUTOTUNE)\n .batch(CONFIG['BATCH_SIZE'])\n .prefetch(AUTOTUNE)\n)\n\nvalidloader = (\n validloader\n .map(load_frame, num_parallel_calls=AUTOTUNE)\n .batch(CONFIG['BATCH_SIZE'])\n .prefetch(AUTOTUNE)\n)","metadata":{"execution":{"iopub.status.busy":"2021-07-21T22:06:35.059668Z","iopub.execute_input":"2021-07-21T22:06:35.059975Z","iopub.status.idle":"2021-07-21T22:06:35.133734Z","shell.execute_reply.started":"2021-07-21T22:06:35.059948Z","shell.execute_reply":"2021-07-21T22:06:35.132946Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# test out the trainloader\nframes, labels = next(iter(trainloader))","metadata":{"execution":{"iopub.status.busy":"2021-07-21T22:06:37.585625Z","iopub.execute_input":"2021-07-21T22:06:37.585941Z","iopub.status.idle":"2021-07-21T22:06:38.176088Z","shell.execute_reply.started":"2021-07-21T22:06:37.585914Z","shell.execute_reply":"2021-07-21T22:06:38.175197Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In order to visualize the samples from our trainloader, I am using W&B. I find it easier to log everything onto W&B to visualize data than to write Matplotlib code. ","metadata":{}},{"cell_type":"code","source":"run = wandb.init(project='brain-tumor-video', job_type='dataloader-viz')\n\nos.makedirs('gifs/')\nfor i, frame in enumerate(frames):\n imageio.mimsave(f'gifs/out_{i}.gif', (frame*255).numpy().astype('uint8')) \n\nwandb.log({'examples': [wandb.Image(f'gifs/out_{i}.gif', caption=f'{label.numpy()}') for i, label in enumerate(labels)]})\n \nrun.finish()","metadata":{"execution":{"iopub.status.busy":"2021-07-21T22:08:33.481909Z","iopub.execute_input":"2021-07-21T22:08:33.482251Z","iopub.status.idle":"2021-07-21T22:08:43.135099Z","shell.execute_reply.started":"2021-07-21T22:08:33.482222Z","shell.execute_reply":"2021-07-21T22:08:43.133981Z"},"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"MRI Sequences, where each sequence is `NUM_FRAMES` long. \n\n","metadata":{}},{"cell_type":"markdown","source":"# š Model\n\nIn order to model both spatial and temporal nature of videos, we can use a hybrid of CNN + LSTM model. \n\n* The `FeatureExtractor` model uses an EfficientNetB0 model as CNN backbone. It will be used to model the spatial aspect of videos.
\n* The `MRIModel` uses a `TimeDistributed` layer that runs the `FeatureExtractor` `NUM_FRAMES` times to get a vector of `(NUM_FRAMES, 1280)`.
\n* This is then fed to a single LSTM layer. You can use GRU and even Transformer in place of LSTM. I have used 256 units as it gave me the best results. ","metadata":{}},{"cell_type":"code","source":"def FeatureExtractor():\n base_model = tf.keras.applications.EfficientNetB0(include_top=False, weights='imagenet')\n base_model.trainabe = True\n\n inputs = Input((CONFIG['IMG_SIZE'], CONFIG['IMG_SIZE'], 1))\n x = Conv2D(3, kernel_size=(3, 3), padding='same', activation='relu')(inputs)\n x = base_model(x, training=True)\n flattened_output = GlobalAveragePooling2D()(x)\n \n return Model(inputs, flattened_output)\n\ntf.keras.backend.clear_session()\nmodel = FeatureExtractor()\nmodel.summary()","metadata":{"execution":{"iopub.status.busy":"2021-07-21T22:20:24.340194Z","iopub.execute_input":"2021-07-21T22:20:24.340528Z","iopub.status.idle":"2021-07-21T22:20:27.150360Z","shell.execute_reply.started":"2021-07-21T22:20:24.340493Z","shell.execute_reply":"2021-07-21T22:20:27.149597Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def MRIModel():\n inputs = Input((CONFIG['NUM_FRAMES'], CONFIG['IMG_SIZE'], CONFIG['IMG_SIZE'], 1))\n feature_extractor = FeatureExtractor()\n \n time_wrapper = TimeDistributed(feature_extractor)(inputs)\n \n lstm_out = LSTM(CONFIG['LSTM_UNITS'], return_sequences=True, name=\"lstm\")(time_wrapper)\n outputs = Dense(1, activation='sigmoid', name=\"lstm_sigmoid\")(lstm_out)\n \n return Model(inputs, outputs)\n\ntf.keras.backend.clear_session() \nmodel = MRIModel()\nmodel.summary()","metadata":{"execution":{"iopub.status.busy":"2021-07-21T22:20:45.217236Z","iopub.execute_input":"2021-07-21T22:20:45.217694Z","iopub.status.idle":"2021-07-21T22:20:49.072781Z","shell.execute_reply.started":"2021-07-21T22:20:45.217653Z","shell.execute_reply":"2021-07-21T22:20:49.071986Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# š
Train\n\nThis is a simple training pipeline that uses early stopping as regularizer and `WandbCallback` to log the metrics to Weights and Biases.","metadata":{}},{"cell_type":"code","source":"# Callbacks\nearlystopper = tf.keras.callbacks.EarlyStopping(\n monitor='val_loss', patience=5, verbose=0, mode='min',\n restore_best_weights=True\n)","metadata":{"execution":{"iopub.status.busy":"2021-07-21T22:21:45.900249Z","iopub.execute_input":"2021-07-21T22:21:45.900584Z","iopub.status.idle":"2021-07-21T22:21:45.904493Z","shell.execute_reply.started":"2021-07-21T22:21:45.900537Z","shell.execute_reply":"2021-07-21T22:21:45.903679Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tf.keras.backend.clear_session() \nmodel = MRIModel()\nmodel.compile('adam', 'binary_crossentropy', metrics=['acc'])\n\nrun = wandb.init(project='brain-tumor-video', \n group='EffnetB0-LSTM-256', \n job_type='train', \n config=CONFIG)\n\n# Train\n_ = model.fit(trainloader, \n epochs=CONFIG['EPOCHS'],\n validation_data=validloader,\n callbacks=[WandbCallback(),\n earlystopper])\n\n# Evaluate\nloss, acc = model.evaluate(validloader)\nwandb.log({'Val Accuracy': round(acc, 3)})\n\nrun.finish()","metadata":{"execution":{"iopub.status.busy":"2021-07-21T22:22:43.644101Z","iopub.execute_input":"2021-07-21T22:22:43.644442Z","iopub.status.idle":"2021-07-21T22:40:18.240837Z","shell.execute_reply.started":"2021-07-21T22:22:43.644411Z","shell.execute_reply":"2021-07-21T22:40:18.239601Z"},"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### [Check out the W&B Dashboard $\\rightarrow$](https://wandb.ai/ayush-thakur/brain-tumor-video?workspace=user-ayush-thakur)\n\n\n\nI have tried out three different LSTM units - 128, 256 and 512. \n\nYou can see that the training curve is not stable. This kernel is a work in progress but wanted to share the idea with wider audience and see if it's a feasible idea to pursue. ","metadata":{}},{"cell_type":"markdown","source":"# WORK IN PROGRESS","metadata":{}}]}