{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 🧠 Tumor Classif. with PyTorch⚡Lightning & MONAI ResNet 3D\n\nThe goal of this challenge is to Predict the status of a genetic biomarker important for brain cancer treatment.\n\nAll the code is refered from public repository: https://github.com/Borda/kaggle_brain-tumor-3D\nAny nice contribution is welcome!","metadata":{}},{"cell_type":"code","source":"! cp -r /kaggle/input/brain-tumor-classif-submissions/package_freeze package_freeze\n\nimport glob, os, shutil\n# finad all packages\npkgs = glob.glob(\"package_freeze/*.xyz\")\n# rename them back to correct name format\n[shutil.move(p, p.replace(\".xyz\", \".tar.gz\")) for p in pkgs]\n\n! pip install -q \"kaggle-brain3D\" --no-index --find-link /kaggle/working/package_freeze\n\n# ! pip install -q https://github.com/Borda/kaggle_brain-tumor-3D/archive/refs/heads/main.zip\n\n! pip uninstall -q -y wandb\n! pip list | grep torch","metadata":{"execution":{"iopub.status.busy":"2021-10-16T11:27:00.273322Z","iopub.execute_input":"2021-10-16T11:27:00.274192Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"! ls -l /kaggle/input/rsna-miccai-brain-tumor-radiogenomic-classification\n! nvidia-smi\n! mkdir /kaggle/temp\n\n%matplotlib inline\n%reload_ext autoreload\n%autoreload 2\n\nimport kaggle_brain3d\nprint(kaggle_brain3d.__version__)","metadata":{"tags":[],"execution":{"iopub.status.busy":"2021-10-11T18:23:34.895081Z","iopub.execute_input":"2021-10-11T18:23:34.89533Z","iopub.status.idle":"2021-10-11T18:23:36.984646Z","shell.execute_reply.started":"2021-10-11T18:23:34.895302Z","shell.execute_reply":"2021-10-11T18:23:36.983906Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data exploration\n\nThese 3 cohorts are structured as follows: Each independent case has a dedicated folder identified by a five-digit number.\nWithin each of these “case” folders, there are four sub-folders, each of them corresponding to each of the structural multi-parametric MRI (mpMRI) scans, in DICOM format.\nThe exact mpMRI scans included are:\n\n- **FLAIR**: Fluid Attenuated Inversion Recovery\n- **T1w**: T1-weighted pre-contrast\n- **T1Gd**: T1-weighted post-contrast\n- **T2w**: T2-weighted\n\n#### according to https://www.aapm.org/meetings/amos2/pdf/34-8205-79886-720.pdf\n\n- T1: weighting weighting better deliniates deliniates anatomy anatomy\n- T2: weighting weighting naturally naturally shows pathology\n\n#### according to https://radiopaedia.org/articles/fluid-attenuated-inversion-recovery\n\nFluid attenuated inversion recovery (FLAIR) is a special inversion recovery sequence with a long inversion time. This removes signal from the cerebrospinal fluid in the resulting images 1. Brain tissue on FLAIR images appears similar to T2 weighted images with grey matter brighter than white matter but CSF is dark instead of bright.\n\nTo null the signal from fluid, the inversion time (TI) of the FLAIR pulse sequence is adjusted such that at equilibrium there is no net transverse magnetization of fluid.\n\nThe FLAIR sequence is part of almost all protocols for imaging the brain, particularly useful in the detection of subtle changes at the periphery of the hemispheres and in the periventricular region close to CSF.","metadata":{}},{"cell_type":"code","source":"import os\nimport glob\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\n\nPATH_DATASET = \"/kaggle/input/rsna-miccai-brain-tumor-radiogenomic-classification\"\nPATH_MODELS = \"../input/meidcalnet-pretrained-3d-resnet-weights\"\nPATH_TEMP = \"/kaggle/temp\"\nSCAN_TYPES = (\"FLAIR\", \"T1w\", \"T1CE\", \"T2w\")\n\ndf_train = pd.read_csv(os.path.join(PATH_DATASET, \"train_labels.csv\"))\ndf_train[\"BraTS21ID\"] = df_train[\"BraTS21ID\"].apply(lambda i: \"%05d\" % i)\ndisplay(df_train.head())","metadata":{"tags":[],"execution":{"iopub.status.busy":"2021-10-11T18:23:36.987386Z","iopub.execute_input":"2021-10-11T18:23:36.987591Z","iopub.status.idle":"2021-10-11T18:23:37.043416Z","shell.execute_reply.started":"2021-10-11T18:23:36.987567Z","shell.execute_reply":"2021-10-11T18:23:37.042734Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"See the dataset label distribution","metadata":{}},{"cell_type":"code","source":"_= df_train[\"MGMT_value\"].value_counts().plot(kind=\"pie\", title=\"label distribution\", autopct=\"%.1f%%\")","metadata":{"tags":[],"execution":{"iopub.status.busy":"2021-10-11T18:23:37.046699Z","iopub.execute_input":"2021-10-11T18:23:37.046912Z","iopub.status.idle":"2021-10-11T18:23:37.217813Z","shell.execute_reply.started":"2021-10-11T18:23:37.046889Z","shell.execute_reply":"2021-10-11T18:23:37.217144Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For almost all scans we have all four types","metadata":{}},{"cell_type":"code","source":"scans = [os.path.basename(p) for p in glob.glob(os.path.join(PATH_DATASET, \"train\", \"*\", \"*\"))]\n_= pd.Series(scans).value_counts().plot(kind=\"bar\", grid=True)","metadata":{"execution":{"iopub.status.busy":"2021-10-11T18:23:37.220399Z","iopub.execute_input":"2021-10-11T18:23:37.220596Z","iopub.status.idle":"2021-10-11T18:23:38.305732Z","shell.execute_reply.started":"2021-10-11T18:23:37.220573Z","shell.execute_reply":"2021-10-11T18:23:38.305039Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Interactive view\n\nshowing particular scan in XYZ dimension/slices","metadata":{}},{"cell_type":"code","source":"from ipywidgets import interact, IntSlider\n\nfrom kaggle_brain3d.utils import load_volume, interpolate_volume, show_volume\nfrom kaggle_brain3d.transforms import crop_volume\n\n\ndef interactive_show(volume_path: str, crop_thr: float):\n    print(f\"loading: {volume_path}\")\n    volume = load_volume(volume_path, percentile=0)\n    print(f\"sample shape: {volume.shape} >> {volume.dtype}\")\n    volume = interpolate_volume(volume)\n    print(f\"interp shape: {volume.shape} >> {volume.dtype}\")\n    volume = crop_volume(volume, crop_thr)\n    print(f\"crop shape: {volume.shape} >> {volume.dtype}\")\n    vol_shape = volume.shape\n    interact(\n        lambda x, y, z: plt.show(show_volume(volume, x, y, z)),\n        x=IntSlider(min=0, max=vol_shape[0], step=5, value=int(vol_shape[0] / 2)),\n        y=IntSlider(min=0, max=vol_shape[1], step=5, value=int(vol_shape[1] / 2)),\n        z=IntSlider(min=0, max=vol_shape[2], step=5, value=int(vol_shape[2] / 2)),\n    )\n\n\nPATH_SAMPLE_VOLUME = os.path.join(PATH_DATASET, \"train\", \"00005\", \"FLAIR\")\n\ninteractive_show(PATH_SAMPLE_VOLUME, crop_thr=1e-6)","metadata":{"tags":[],"execution":{"iopub.status.busy":"2021-10-11T18:23:38.307037Z","iopub.execute_input":"2021-10-11T18:23:38.307294Z","iopub.status.idle":"2021-10-11T18:24:17.309122Z","shell.execute_reply.started":"2021-10-11T18:23:38.307263Z","shell.execute_reply":"2021-10-11T18:24:17.308383Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Prepare dataset\n\n### Pytorch Dataset\n\nThe basic building block is traforming raw data to Torch Dataset.\nWe have here loading particular DICOM images into a volume and saving as temp/cacher, so we do not need to take the very time demanding loading do next time - this boost the IO from about 2h to 8min\n\nAt the end we show a few sample images from prepared dataset.","metadata":{}},{"cell_type":"code","source":"import os\nimport pandas as pd\nimport torch\nfrom tqdm.auto import tqdm\n\nfrom kaggle_brain3d.data import BrainScansDataset\nfrom kaggle_brain3d.transforms import resize_volume\n\n# ==============================\n\nds = BrainScansDataset(\n    image_dir=os.path.join(PATH_DATASET, \"train\"),\n    df_table=os.path.join(PATH_DATASET, \"train_labels.csv\"),\n    crop_thr=None, cache_dir=PATH_TEMP,\n)\nfor i in tqdm(range(2)):\n    img = ds[i * 10][\"data\"]\n    img = resize_volume(img[0])\n    show_volume(img, fig_size=(9, 6))","metadata":{"tags":[],"execution":{"iopub.status.busy":"2021-10-11T18:24:17.310382Z","iopub.execute_input":"2021-10-11T18:24:17.310859Z","iopub.status.idle":"2021-10-11T18:25:09.45722Z","shell.execute_reply.started":"2021-10-11T18:24:17.31082Z","shell.execute_reply":"2021-10-11T18:25:09.456531Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Lightning DataModule\n\nIt is constric to wrap all data-related peaces and define Pytoch dataloder for Training / Validation / Testing phase.\n\nAt the end we show a few sample images from the fost training batch.","metadata":{}},{"cell_type":"code","source":"from functools import partial\nimport rising.transforms as rtr\nfrom rising.loading import DataLoader, default_transform_call\nfrom rising.random import DiscreteParameter, UniformParameter\n\nfrom kaggle_brain3d.data import BrainScansDM, TRAIN_TRANSFORMS, VAL_TRANSFORMS\nfrom kaggle_brain3d.transforms import RandomAffine, rising_zero_mean\n\n# ==============================\n\ndm = BrainScansDM(\n    data_dir=PATH_DATASET,\n    scan_types=[\"T2w\"],\n    vol_size=128,\n    crop_thr=None,\n    # crop_thr=1e-6,  # experimental crop threshold\n    # batch_size=4,  # for full model training\n    batch_size=6,  # for finetune head\n    cache_dir=PATH_TEMP,\n    # in_memory=True,\n    num_workers=6,\n    train_transforms=rtr.Compose(TRAIN_TRANSFORMS, transform_call=default_transform_call),\n    valid_transforms=rtr.Compose(VAL_TRANSFORMS, transform_call=default_transform_call),\n)\ndm.prepare_data(num_proc=3)\ndm.setup()\n# dm.prepare_data(num_proc=3, dataset=dm.test_dataset)\nprint(f\"Training batches: {len(dm.train_dataloader())}\")\nprint(f\"Validation batches: {len(dm.val_dataloader())}\")\nprint(f\"Test batches: {len(dm.test_dataloader())}\")\n\n# Quick view\nfor batch in dm.train_dataloader():\n    for i in range(2):\n        show_volume(batch[\"data\"][i][0], fig_size=(6, 4), v_min_max=(-1., 3.))\n    break","metadata":{"tags":[],"execution":{"iopub.status.busy":"2021-10-11T18:25:09.458648Z","iopub.execute_input":"2021-10-11T18:25:09.458943Z","iopub.status.idle":"2021-10-11T20:01:35.476216Z","shell.execute_reply.started":"2021-10-11T18:25:09.458909Z","shell.execute_reply":"2021-10-11T20:01:35.475464Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Prepare 3D model\n\nLightningModule is the core of PL, it wrappes all model related peaces, mainly:\n\n- the model/architecture/weights\n- evaluation metrics\n- configs for optimizer and LR scheduler","metadata":{}},{"cell_type":"code","source":"from torchsummary import summary\nfrom kaggle_brain3d.models import LitBrainMRI, create_pretrained_medical_resnet\nfrom monai.networks.nets import resnet10, resnet18, resnet34, resnet50, SEResNet50\nfrom torch.optim import SGD, ASGD, SparseAdam\n\n# ==============================\n\n\nPATH_PRETRAINED_WEIGHTS = os.path.join(PATH_MODELS, \"resnet_18_23dataset.pth\")\nnet, pretraineds_layers = create_pretrained_medical_resnet(PATH_PRETRAINED_WEIGHTS, model_constructor=resnet18)\n\n# net = SEResNet50(spatial_dims=3, in_channels=1, pretrained=True, num_classes=2)\n\nmodel = LitBrainMRI(net=net, pretrained_params=pretraineds_layers, lr=5e-4)\n# summary(model, input_size=(1, 128, 128, 128))","metadata":{"tags":[],"execution":{"iopub.status.busy":"2021-10-11T20:01:35.477673Z","iopub.execute_input":"2021-10-11T20:01:35.477968Z","iopub.status.idle":"2021-10-11T20:01:48.35883Z","shell.execute_reply.started":"2021-10-11T20:01:35.477927Z","shell.execute_reply":"2021-10-11T20:01:48.357129Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Train a model\n\nLightning forces the following structure to your code which makes it reusable and shareable:\n\n- Research code (the LightningModule).\n- Engineering code (you delete, and is handled by the Trainer).\n- Non-essential research code (logging, etc... this goes in Callbacks).\n- Data (use PyTorch DataLoaders or organize them into a LightningDataModule).\n\nOnce you do this, you can train on multiple-GPUs, TPUs, CPUs and even in 16-bit precision without changing your code!","metadata":{}},{"cell_type":"code","source":"import pytorch_lightning as pl\nfrom kaggle_brain3d.models import FineTuneCB\n\nlogger = pl.loggers.CSVLogger(save_dir='logs/', name=model.name)\nswa = pl.callbacks.StochasticWeightAveraging(swa_epoch_start=0.6)\nfine = FineTuneCB(unfreeze_epoch=20)\nckpt = pl.callbacks.ModelCheckpoint(\n    monitor='valid/auroc',\n    save_top_k=1,\n    save_last=True,\n    filename='checkpoint/{epoch:02d}-{valid/auroc:.4f}',\n    mode='max',\n)\n\n# ==============================\n\ntrainer = pl.Trainer(\n    # overfit_batches=5,\n    # fast_dev_run=True,\n    gpus=1,\n    callbacks=[ckpt],  #, fine, swa\n    logger=logger,\n    max_epochs=25,\n    precision=16,\n    benchmark=True,\n    accumulate_grad_batches=8,\n    # val_check_interval=0.5,\n    progress_bar_refresh_rate=1,\n    log_every_n_steps=5,\n    weights_summary='top',\n    auto_lr_find=True,\n#     auto_scale_batch_size='binsearch',\n)\n\n# ==============================\n\n# trainer.tune(\n#     model, \n#     datamodule=dm, \n#     lr_find_kwargs=dict(min_lr=2e-5, max_lr=1e-3, num_training=10),\n#     # scale_batch_size_kwargs=dict(max_trials=5),\n# )\n# print(f\"Batch size: {dm.batch_size}\")\n# print(f\"Learning Rate: {model.learning_rate}\")\n\n# ==============================\n\ntrainer.fit(model=model, datamodule=dm)","metadata":{"tags":[],"execution":{"iopub.status.busy":"2021-10-11T20:01:48.361479Z","iopub.execute_input":"2021-10-11T20:01:48.361918Z","iopub.status.idle":"2021-10-11T20:50:30.491122Z","shell.execute_reply.started":"2021-10-11T20:01:48.36188Z","shell.execute_reply":"2021-10-11T20:50:30.489175Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Training progress","metadata":{}},{"cell_type":"code","source":"metrics = pd.read_csv(f'{logger.log_dir}/metrics.csv')\ndisplay(metrics.head())\n\naggreg_metrics = []\nagg_col = \"epoch\"\nfor i, dfg in metrics.groupby(agg_col):\n    agg = dict(dfg.mean())\n    agg[agg_col] = i\n    aggreg_metrics.append(agg)\n\ndf_metrics = pd.DataFrame(aggreg_metrics)\ndf_metrics[['train/loss', 'valid/loss']].plot(grid=True, legend=True, xlabel=agg_col)\ndf_metrics[['train/f1', 'train/auroc', 'valid/f1', 'valid/auroc']].plot(grid=True, legend=True, xlabel=agg_col)","metadata":{"execution":{"iopub.status.busy":"2021-10-11T20:50:30.49204Z","iopub.status.idle":"2021-10-11T20:50:30.492348Z","shell.execute_reply.started":"2021-10-11T20:50:30.492192Z","shell.execute_reply":"2021-10-11T20:50:30.492214Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Predictions","metadata":{}},{"cell_type":"code","source":"model.eval()\nmodel.cpu()\nsubmission = []\n\nfor batch in dm.test_dataloader():\n    print(batch.keys())\n    print(batch.get(\"label\"))\n    imgs = batch.get(\"data\")\n    print(imgs.shape)\n    with torch.no_grad():\n        preds = model(imgs)\n    print(preds)\n    probs = torch.nn.functional.softmax(preds)\n    print(probs)\n    break","metadata":{"execution":{"iopub.status.busy":"2021-10-11T20:50:30.493763Z","iopub.status.idle":"2021-10-11T20:50:30.494173Z","shell.execute_reply.started":"2021-10-11T20:50:30.493943Z","shell.execute_reply":"2021-10-11T20:50:30.493963Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from kaggle_brain3d.models import make_submission\n\ndm.batch_size = 2\ndf_submission = make_submission(model, dm.test_dataloader(), \"cuda\" if torch.cuda.is_available() else \"cpu\")\ndisplay(df_submission)\ndf_submission[\"MGMT_value\"].to_csv(\"submission.csv\")","metadata":{"execution":{"iopub.status.busy":"2021-10-11T20:50:30.495215Z","iopub.status.idle":"2021-10-11T20:50:30.495619Z","shell.execute_reply.started":"2021-10-11T20:50:30.495397Z","shell.execute_reply":"2021-10-11T20:50:30.495418Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_submission[[\"MGMT_value\"]].hist(bins=25)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"! cat submission.csv","metadata":{},"execution_count":null,"outputs":[]}]}