{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## This Kernel is a fork of [Efficientnet3D with one MRI type](https://www.kaggle.com/rluethy/efficientnet3d-with-one-mri-type)\n\nAs many others have said during the course of this competition, most models aren't learning anything. \n1. Be it the difference between the [**Public Leaderboard and Local CV AUC**](https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/255352)\n2. [**Models failing to generalize**](https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/266173)\n3. [**Models not training**](https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/265777) in the first place","metadata":{}},{"cell_type":"markdown","source":"## [**Weights and Biases Report ⭐️**](https://wandb.ai/sauravmaheshkar/RSNA-MICCAI/reports/The-Fluke--VmlldzoxMDA2MDQy) | [**Weights and Biases Project**](https://wandb.ai/sauravmaheshkar/RSNA-MICCAI)","metadata":{}},{"cell_type":"markdown","source":"The motivation for these experiments come from [**Chai Time Kaggle Talks with Anjum Sayed (Datasaurus)**](https://youtu.be/udw-uSV66EQ) Video on the [**Weights and Biases Channel**](https://www.youtube.com/WeightsBiases). Anjum mentioned that a good way to check if the models are learning anything is to just change the random seeds and see if it affects the performance.\n\n[![Video Title](https://api.wandb.ai/files/sauravmaheshkar/images/projects/436131/cc7a6207.png)](https://youtu.be/udw-uSV66EQ)","metadata":{}},{"cell_type":"markdown","source":"# Models Don't Learn 🤷🏻","metadata":{}},{"cell_type":"markdown","source":"![](https://raw.githubusercontent.com/SauravMaheshkar/RSNA-MICCAI/main/assets/Fluke-Training-Loss.svg)\n\n![](https://raw.githubusercontent.com/SauravMaheshkar/RSNA-MICCAI/main/assets/Fluke-Validation-Loss.svg)","metadata":{}},{"cell_type":"code","source":"%%capture\nimport os\nimport sys \nimport json\nimport glob\nimport random\nimport collections\nimport time\nimport re\n\nimport numpy as np\nimport pandas as pd\nimport pydicom\nfrom pydicom.pixel_data_handlers.util import apply_voi_lut\nimport cv2\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nimport torch\nfrom torch import nn\nfrom torch.utils import data as torch_data\nfrom sklearn import model_selection as sk_model_selection\nfrom torch.nn import functional as torch_functional\nimport torch.nn.functional as F\n\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.metrics import roc_auc_score\n\n# Install Weights and Biases \n!pip3 install -r ../input/rsnawrapper/requirements.txt\n\n# Weights and Biases Setup\nimport wandb\nfrom wandb.keras import WandbCallback\nfrom kaggle_secrets import UserSecretsClient\nuser_secrets = UserSecretsClient()\napi_key = user_secrets.get_secret(\"WANDB_API_KEY\")\nwandb.login(key=api_key);\n\nfrom rsna.utils import load_dicom_images_3d\na = load_dicom_images_3d(\"00000\")\nprint(a.shape)\nprint(np.min(a), np.max(a), np.mean(a), np.median(a))","metadata":{"papermill":{"duration":1.048295,"end_time":"2021-07-14T20:26:46.309722","exception":false,"start_time":"2021-07-14T20:26:45.261427","status":"completed"},"tags":[],"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-09-03T03:17:12.47052Z","iopub.execute_input":"2021-09-03T03:17:12.470946Z","iopub.status.idle":"2021-09-03T03:18:45.006467Z","shell.execute_reply.started":"2021-09-03T03:17:12.470835Z","shell.execute_reply":"2021-09-03T03:18:45.005376Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if os.path.exists(\"../input/rsna-miccai-brain-tumor-radiogenomic-classification\"):\n    data_directory = '../input/rsna-miccai-brain-tumor-radiogenomic-classification'\n    pytorch3dpath = \"../input/efficientnetpyttorch3d/EfficientNet-PyTorch-3D\"\n    wrapperdir = \"../input/rsnawrapper\"\n    \nmri_types = ['FLAIR','T1w','T1wCE','T2w']\nSIZE = 256\nNUM_IMAGES = 64\n\nsys.path.append(pytorch3dpath)\nsys.path.append(wrapperdir)\nfrom efficientnet_pytorch_3d import EfficientNet3D","metadata":{"lines_to_end_of_cell_marker":2,"lines_to_next_cell":2,"papermill":{"duration":0.05565,"end_time":"2021-07-14T20:26:46.486521","exception":false,"start_time":"2021-07-14T20:26:46.430871","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-03T03:21:31.563554Z","iopub.execute_input":"2021-09-03T03:21:31.563952Z","iopub.status.idle":"2021-09-03T03:21:31.605698Z","shell.execute_reply.started":"2021-09-03T03:21:31.563919Z","shell.execute_reply":"2021-09-03T03:21:31.604974Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Random Seed 🌱","metadata":{}},{"cell_type":"code","source":"def set_seed(seed):\n    random.seed(seed)\n    os.environ[\"PYTHONHASHSEED\"] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    if torch.cuda.is_available():\n        torch.cuda.manual_seed_all(seed)\n        torch.backends.cudnn.deterministic = True\n\nset_seed(21)","metadata":{"papermill":{"duration":0.668331,"end_time":"2021-07-14T20:27:48.114522","exception":false,"start_time":"2021-07-14T20:27:47.446191","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-03T03:21:46.891349Z","iopub.execute_input":"2021-09-03T03:21:46.891711Z","iopub.status.idle":"2021-09-03T03:21:46.901168Z","shell.execute_reply.started":"2021-09-03T03:21:46.891681Z","shell.execute_reply":"2021-09-03T03:21:46.900035Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# ✂️ Train/Test Split","metadata":{}},{"cell_type":"code","source":"train_df = pd.read_csv(f\"{data_directory}/train_labels.csv\")\ndisplay(train_df)\n\ndf_train, df_valid = sk_model_selection.train_test_split(\n    train_df, \n    test_size=0.2, \n    random_state=12, \n    stratify=train_df[\"MGMT_value\"],\n)","metadata":{"papermill":{"duration":0.633753,"end_time":"2021-07-14T20:27:49.350524","exception":false,"start_time":"2021-07-14T20:27:48.716771","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-03T03:21:48.75736Z","iopub.execute_input":"2021-09-03T03:21:48.758034Z","iopub.status.idle":"2021-09-03T03:21:48.832345Z","shell.execute_reply.started":"2021-09-03T03:21:48.757835Z","shell.execute_reply":"2021-09-03T03:21:48.831074Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 💿 Dataset","metadata":{}},{"cell_type":"code","source":"# Boiler Plate code from the library\nfrom rsna.dataloader import Dataset","metadata":{"papermill":{"duration":0.634322,"end_time":"2021-07-14T20:27:50.594701","exception":false,"start_time":"2021-07-14T20:27:49.960379","status":"completed"},"tags":[],"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-09-03T03:23:45.215118Z","iopub.execute_input":"2021-09-03T03:23:45.215544Z","iopub.status.idle":"2021-09-03T03:23:45.221309Z","shell.execute_reply.started":"2021-09-03T03:23:45.21551Z","shell.execute_reply":"2021-09-03T03:23:45.219805Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 🏠 Model Class","metadata":{}},{"cell_type":"code","source":"# Boiler Plate code from the library\nfrom rsna.nn import Model","metadata":{"papermill":{"duration":0.825458,"end_time":"2021-07-14T20:27:55.604161","exception":false,"start_time":"2021-07-14T20:27:54.778703","status":"completed"},"tags":[],"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-09-03T03:23:11.484664Z","iopub.execute_input":"2021-09-03T03:23:11.485424Z","iopub.status.idle":"2021-09-03T03:23:11.501824Z","shell.execute_reply.started":"2021-09-03T03:23:11.485364Z","shell.execute_reply":"2021-09-03T03:23:11.500546Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 💪🏻 Trainer Class","metadata":{}},{"cell_type":"code","source":"# Boiler Plate code from the library\nfrom rsna.engine import Trainer","metadata":{"papermill":{"duration":0.637077,"end_time":"2021-07-14T20:27:58.09407","exception":false,"start_time":"2021-07-14T20:27:57.456993","status":"completed"},"tags":[],"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-09-03T03:23:07.729143Z","iopub.execute_input":"2021-09-03T03:23:07.729587Z","iopub.status.idle":"2021-09-03T03:23:07.735341Z","shell.execute_reply.started":"2021-09-03T03:23:07.729554Z","shell.execute_reply":"2021-09-03T03:23:07.733812Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 🔥 Training","metadata":{}},{"cell_type":"code","source":"device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n\nCONFIG = dict(\n    BATCH_SIZE = 4,\n    EPOCHS = 10,\n    PATIENCE = 10,\n    SIZE = 256,\n    NUM_IMAGES = 64,\n    competition = 'rsna-miccai-brain',\n    _wandb_kernel = 'sauravm'\n)\n\ndef train_mri_type(df_train, df_valid, mri_type):\n    if mri_type==\"all\":\n        train_list = []\n        valid_list = []\n        for mri_type in mri_types:\n            df_train.loc[:,\"MRI_Type\"] = mri_type\n            train_list.append(df_train.copy())\n            df_valid.loc[:,\"MRI_Type\"] = mri_type\n            valid_list.append(df_valid.copy())\n\n        df_train = pd.concat(train_list)\n        df_valid = pd.concat(valid_list)\n    else:\n        df_train.loc[:,\"MRI_Type\"] = mri_type\n        df_valid.loc[:,\"MRI_Type\"] = mri_type\n\n    print(df_train.shape, df_valid.shape)\n    display(df_train.head())\n    \n    train_data_retriever = Dataset(\n        df_train[\"BraTS21ID\"].values, \n        df_train[\"MGMT_value\"].values, \n        df_train[\"MRI_Type\"].values,\n        augment=False\n    )\n\n    valid_data_retriever = Dataset(\n        df_valid[\"BraTS21ID\"].values, \n        df_valid[\"MGMT_value\"].values,\n        df_valid[\"MRI_Type\"].values\n    )\n\n    train_loader = torch_data.DataLoader(\n        train_data_retriever,\n        batch_size=CONFIG['BATCH_SIZE'],\n        shuffle=True,\n        num_workers=8,\n    )\n\n    valid_loader = torch_data.DataLoader(\n        valid_data_retriever, \n        batch_size=CONFIG['BATCH_SIZE'],\n        shuffle=False,\n        num_workers=8,\n    )\n    \n    run = wandb.init(project='RSNA-MICCAI', \n                     entity='sauravmaheshkar', \n                     group='4-types', \n                     job_type='train', \n                     config=CONFIG)\n\n    model = Model(model_name = \"efficientnet-b0\")\n    model.to(device)\n\n    wandb.watch(model)\n    \n    optimizer = torch.optim.AdamW(model.parameters(), lr=0.001)\n\n    criterion = torch_functional.binary_cross_entropy_with_logits\n\n    trainer = Trainer(\n        model, \n        device, \n        optimizer, \n        criterion\n    )\n\n    history = trainer.fit(\n        CONFIG['EPOCHS'], \n        train_loader, \n        valid_loader, \n        f\"{mri_type}\", \n        CONFIG['PATIENCE'],\n    )\n    \n    run.finish()\n    \n    return trainer.lastmodel\n\nmodelfiles = None\n\n\nif not modelfiles:\n    modelfiles = [train_mri_type(df_train, df_valid, m) for m in mri_types]\n    print(modelfiles)","metadata":{"lines_to_next_cell":2,"papermill":{"duration":447.387602,"end_time":"2021-07-14T20:35:26.110421","exception":false,"start_time":"2021-07-14T20:27:58.722819","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# ⚙️ Prediction","metadata":{}},{"cell_type":"code","source":"def predict(modelfile, df, mri_type, split):\n    print(\"Predict:\", modelfile, mri_type, df.shape)\n    df.loc[:,\"MRI_Type\"] = mri_type\n    data_retriever = Dataset(\n        df.index.values, \n        mri_type=df[\"MRI_Type\"].values,\n        split=split\n    )\n\n    data_loader = torch_data.DataLoader(\n        data_retriever,\n        batch_size=CONFIG['BATCH_SIZE'],\n        shuffle=False,\n        num_workers=8,\n    )\n   \n    model = Model()\n    model.to(device)\n    \n    checkpoint = torch.load(modelfile)\n    model.load_state_dict(checkpoint[\"model_state_dict\"])\n    model.eval()\n    \n    y_pred = []\n    ids = []\n\n    for e, batch in enumerate(data_loader,1):\n        print(f\"{e}/{len(data_loader)}\", end=\"\\r\")\n        with torch.no_grad():\n            tmp_pred = torch.sigmoid(model(batch[\"X\"].to(device))).cpu().numpy().squeeze()\n            if tmp_pred.size == 1:\n                y_pred.append(tmp_pred)\n            else:\n                y_pred.extend(tmp_pred.tolist())\n            ids.extend(batch[\"id\"].numpy().tolist())\n            \n    preddf = pd.DataFrame({\"BraTS21ID\": ids, \"MGMT_value\": y_pred}) \n    preddf = preddf.set_index(\"BraTS21ID\")\n    return preddf","metadata":{"papermill":{"duration":0.990911,"end_time":"2021-07-14T20:35:30.482254","exception":false,"start_time":"2021-07-14T20:35:29.491343","status":"completed"},"tags":[],"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-09-02T02:52:44.069769Z","iopub.status.idle":"2021-09-02T02:52:44.070596Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Ensemble for validation","metadata":{}},{"cell_type":"code","source":"df_valid = df_valid.set_index(\"BraTS21ID\")\ndf_valid[\"MGMT_pred\"] = 0\nfor m, mtype in zip(modelfiles,  mri_types):\n    pred = predict(m, df_valid, mtype, \"train\")\n    df_valid[\"MGMT_pred\"] += pred[\"MGMT_value\"]\ndf_valid[\"MGMT_pred\"] /= len(modelfiles)\nauc = roc_auc_score(df_valid[\"MGMT_value\"], df_valid[\"MGMT_pred\"])\nprint(f\"Validation ensemble AUC: {auc:.4f}\")\nsns.displot(df_valid[\"MGMT_pred\"])","metadata":{"execution":{"iopub.status.busy":"2021-09-02T02:52:44.071815Z","iopub.status.idle":"2021-09-02T02:52:44.072516Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Ensemble for submission","metadata":{}},{"cell_type":"code","source":"submission = pd.read_csv(f\"{data_directory}/sample_submission.csv\", index_col=\"BraTS21ID\")\n\nsubmission[\"MGMT_value\"] = 0\nfor m, mtype in zip(modelfiles, mri_types):\n    pred = predict(m, submission, mtype, split=\"test\")\n    submission[\"MGMT_value\"] += pred[\"MGMT_value\"]\n\nsubmission[\"MGMT_value\"] /= len(modelfiles)\nsubmission[\"MGMT_value\"].to_csv(\"submission.csv\")","metadata":{"papermill":{"duration":0.990911,"end_time":"2021-07-14T20:35:30.482254","exception":false,"start_time":"2021-07-14T20:35:29.491343","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-02T02:52:44.073616Z","iopub.status.idle":"2021-09-02T02:52:44.074304Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission","metadata":{"papermill":{"duration":0.972383,"end_time":"2021-07-14T20:35:41.939464","exception":false,"start_time":"2021-07-14T20:35:40.967081","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-02T02:52:44.075343Z","iopub.status.idle":"2021-09-02T02:52:44.076052Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.displot(submission[\"MGMT_value\"])","metadata":{"execution":{"iopub.status.busy":"2021-09-02T02:52:44.077085Z","iopub.status.idle":"2021-09-02T02:52:44.077785Z"},"trusted":true},"execution_count":null,"outputs":[]}]}