{"metadata":{"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":71549,"databundleVersionId":8561470,"sourceType":"competition"},{"sourceId":183439689,"sourceType":"kernelVersion"}],"dockerImageVersionId":30733,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# RSNA2024 LSDC Training DenseNet\nThis notebook is forked [here](https://www.kaggle.com/code/itsuki9180/rsna2024-lsdc-training-baseline). In the [previous notebook](https://www.kaggle.com/code/itsuki9180/rsna2024-lsdc-making-dataset), the author selected the images we wanted to use and exported them to png.The original notebook training. And the original other trained EfficientNet_B4 model with these images. \n\nI desided to change the model to see if there is any improvement. The reason why I choose DenseNet201 for training is that DenseNet201 has generally same number of parameters and size with EfficientNet_B4 so that I believe that Kaggle GPU could handle it and we don't need extra machine.\n\n### Credit\n- [RSNA2024 LSDC Making Dataset](https://www.kaggle.com/code/itsuki9180/rsna2024-lsdc-making-dataset) \n- [RSNA2024 LSDC Training DenseNet](https://www.kaggle.com/code/hugowjd/rsna2024-lsdc-training-densenet) <- you're reading now\n- [RSNA2024 LSDC Submission DenseNet](https://www.kaggle.com/code/itsuki9180/rsna2024-lsdc-submission-baseline)\n\n### Reference:\n* [Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q. Densely connected convolutional networks. CVPR, 2017.](https://arxiv.org/abs/1608.06993)\n* [Mingxing Tan and Quoc V. Le. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. ICML 2019.](https://arxiv.org/abs/1905.11946)\n\n### Future Improvement\n* I'm certain that we can run other models to train these images. We can do it by changing ***MODEL_NAME*** parameters in **Config** session. I would list some CNN models which are suitable for image classification and these numbers of parameters.\n  * ResNet:\n    * ResNet-18: ~11.7 million parameters\n    * **ResNet-34: ~21.8 million parameters**\n    * **ResNet-50: ~25.6 million parameters**\n    * ResNet-101: ~44.5 million parameters\n    * ResNet-152: ~60 million parameters\n  * VGG:\n    * VGG-16: ~138 million parameters\n    * VGG-19: ~143 million parameters\n  * Inception Networks:\n    * Inception v1 (GoogleNet): ~6.8 million parameters\n    * Inception v3: ~23.8 million parameters\n  * DenseNet:\n    * DenseNet-121: ~8 million parameters\n    * **DenseNet-169: ~14 million parameters**\n    * **DenseNet-201: ~20 million parameters**\n    * DenseNet-264: ~33 million parameters\n  * MobileNets (parameters can vary significantly with changes in alpha and resolution multipliers):\n    * MobileNetV1 (1.0 224): ~4.2 million parameters\n    * MobileNetV2 (1.0 224): ~3.5 million parameters\n    * MobileNetV3 Large: ~5.4 million parameters\n  * Vision Transformers (ViT):\n    * ViT-B/16 (base model with patch size 16x16): ~86 million parameters\n  * Xception:\n    * **Xception: ~22.9 million parameters**\n  * EfficientNet\n    * EfficientNet-B0: ~5.3 million parameters\n    * EfficientNet-B1: ~7.8 million parameters\n    * EfficientNet-B2: ~9.2 million parameters\n    * EfficientNet-B3: ~12 million parameters\n    * **EfficientNet-B4: ~19 million parameters**\n    * EfficientNet-B5: ~30 million parameters\n    * EfficientNet-B6: ~43 million parameters\n    * EfficientNet-B7: ~66 million parameters\n* The original author said that we can improve the dataset making process\n* I only trained 5 **folds** and 10 **epochs**, you can modify these parameters. But take care of overfitting.","metadata":{}},{"cell_type":"markdown","source":"# Import Libralies","metadata":{}},{"cell_type":"code","source":"!unzip -q /kaggle/input/rsna2024-lsdc-making-dataset/_output_.zip ","metadata":{"execution":{"iopub.status.busy":"2024-06-19T21:36:10.216547Z","iopub.execute_input":"2024-06-19T21:36:10.217471Z","iopub.status.idle":"2024-06-19T21:40:14.266145Z","shell.execute_reply.started":"2024-06-19T21:36:10.217434Z","shell.execute_reply":"2024-06-19T21:40:14.264813Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import os\nimport gc\nimport sys\nfrom PIL import Image\nimport cv2\nimport math, random\nimport numpy as np\nimport pandas as pd\nfrom glob import glob\nfrom tqdm import tqdm\nimport matplotlib.pyplot as plt\nfrom sklearn.model_selection import KFold\n\nfrom collections import OrderedDict\n\nimport torch\nimport torch.nn.functional as F\nfrom torch import nn\nfrom torch.utils.data import DataLoader, Dataset\nfrom torch.optim import AdamW\n\nimport timm\nfrom transformers import get_cosine_schedule_with_warmup\n\nimport albumentations as A\n\nfrom sklearn.model_selection import KFold","metadata":{"execution":{"iopub.status.busy":"2024-06-19T21:40:14.268333Z","iopub.execute_input":"2024-06-19T21:40:14.268612Z","iopub.status.idle":"2024-06-19T21:40:22.141987Z","shell.execute_reply.started":"2024-06-19T21:40:14.268587Z","shell.execute_reply":"2024-06-19T21:40:22.141093Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"rd = '/kaggle/input/rsna-2024-lumbar-spine-degenerative-classification'","metadata":{"execution":{"iopub.status.busy":"2024-06-19T21:40:22.143078Z","iopub.execute_input":"2024-06-19T21:40:22.143551Z","iopub.status.idle":"2024-06-19T21:40:22.147711Z","shell.execute_reply.started":"2024-06-19T21:40:22.143524Z","shell.execute_reply":"2024-06-19T21:40:22.146808Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Config","metadata":{}},{"cell_type":"code","source":"# Remember to change NOT_DEBUG to True for real training\nNOT_DEBUG = False # True -> run naormally, False -> debug mode, with lesser computing cost\n\nOUTPUT_DIR = f'/kaggle/working/rsna24-results'\ndevice = 'cuda:0' if torch.cuda.is_available() else 'cpu'\nN_WORKERS = os.cpu_count()\nUSE_AMP = True # can change True if using T4 or newer than Ampere\nSEED = 8620\n\nIMG_SIZE = [512, 512]\nIN_CHANS = 30\nN_LABELS = 25\nN_CLASSES = 3 * N_LABELS\n\nAUG_PROB = 0.75\nSELECTED_FOLDS = [0]\nN_FOLDS = 5 if NOT_DEBUG else 2\nEPOCHS = 10 if NOT_DEBUG else 2\n# MODEL_NAME = \"tf_efficientnet_b4.ns_jft_in1k\" if NOT_DEBUG else \"tf_efficientnet_b0.ns_jft_in1k\"\n# TODO: you can choose other convolutional neural network (CNN) architectures designed to \n#       achieve state-of-the-art accuracy in various computer vision tasks\n\nMODEL_NAME = \"densenet201\" if NOT_DEBUG else \"densenet121\"\n\nGRAD_ACC = 2\nTGT_BATCH_SIZE = 32\nBATCH_SIZE = TGT_BATCH_SIZE // GRAD_ACC\nMAX_GRAD_NORM = None\nEARLY_STOPPING_EPOCH = 3\n\nLR = 2e-4 * TGT_BATCH_SIZE / 32\nWD = 1e-2\nAUG = True","metadata":{"execution":{"iopub.status.busy":"2024-06-19T21:40:22.149849Z","iopub.execute_input":"2024-06-19T21:40:22.150116Z","iopub.status.idle":"2024-06-19T21:40:22.237106Z","shell.execute_reply.started":"2024-06-19T21:40:22.150094Z","shell.execute_reply":"2024-06-19T21:40:22.236241Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"os.makedirs(OUTPUT_DIR, exist_ok=True)","metadata":{"execution":{"iopub.status.busy":"2024-06-19T21:40:22.238514Z","iopub.execute_input":"2024-06-19T21:40:22.238775Z","iopub.status.idle":"2024-06-19T21:40:22.24697Z","shell.execute_reply.started":"2024-06-19T21:40:22.238748Z","shell.execute_reply":"2024-06-19T21:40:22.246226Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def set_random_seed(seed: int = 2222, deterministic: bool = False):\n    \"\"\"Set seeds\"\"\"\n    random.seed(seed)\n    np.random.seed(seed)\n    os.environ[\"PYTHONHASHSEED\"] = str(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)  # type: ignore\n    torch.backends.cudnn.benchmark = True\n    torch.backends.cudnn.deterministic = deterministic  # type: ignore\n\nset_random_seed(SEED)","metadata":{"execution":{"iopub.status.busy":"2024-06-19T21:40:22.248224Z","iopub.execute_input":"2024-06-19T21:40:22.248598Z","iopub.status.idle":"2024-06-19T21:40:22.262432Z","shell.execute_reply.started":"2024-06-19T21:40:22.248569Z","shell.execute_reply":"2024-06-19T21:40:22.261621Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Open Dataframes","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv(f'{rd}/train.csv')\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2024-06-19T21:40:22.263455Z","iopub.execute_input":"2024-06-19T21:40:22.263763Z","iopub.status.idle":"2024-06-19T21:40:22.338513Z","shell.execute_reply.started":"2024-06-19T21:40:22.263733Z","shell.execute_reply":"2024-06-19T21:40:22.337546Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Change the state to Label.\n\nThe dataframe contains some Nans, which we will replace with -100 so that We and function can ignore them when calculating the loss and score.","metadata":{}},{"cell_type":"code","source":"df = df.fillna(-100)","metadata":{"execution":{"iopub.status.busy":"2024-06-19T21:40:22.339611Z","iopub.execute_input":"2024-06-19T21:40:22.339856Z","iopub.status.idle":"2024-06-19T21:40:22.354547Z","shell.execute_reply.started":"2024-06-19T21:40:22.339834Z","shell.execute_reply":"2024-06-19T21:40:22.353766Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"label2id = {'Normal/Mild': 0, 'Moderate':1, 'Severe':2}\ndf = df.replace(label2id)\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2024-06-19T21:40:22.35555Z","iopub.execute_input":"2024-06-19T21:40:22.355813Z","iopub.status.idle":"2024-06-19T21:40:22.414837Z","shell.execute_reply.started":"2024-06-19T21:40:22.355789Z","shell.execute_reply":"2024-06-19T21:40:22.414063Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"CONDITIONS = [\n    'Spinal Canal Stenosis', \n    'Left Neural Foraminal Narrowing', \n    'Right Neural Foraminal Narrowing',\n    'Left Subarticular Stenosis',\n    'Right Subarticular Stenosis'\n]\n\nLEVELS = [\n    'L1/L2',\n    'L2/L3',\n    'L3/L4',\n    'L4/L5',\n    'L5/S1',\n]","metadata":{"execution":{"iopub.status.busy":"2024-06-19T21:40:22.41939Z","iopub.execute_input":"2024-06-19T21:40:22.42012Z","iopub.status.idle":"2024-06-19T21:40:22.42504Z","shell.execute_reply.started":"2024-06-19T21:40:22.420085Z","shell.execute_reply":"2024-06-19T21:40:22.424076Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Define Dataset\n\nThis implementation is very slow and leaves a lot of room for improvement.","metadata":{}},{"cell_type":"code","source":"class RSNA24Dataset(Dataset):\n    def __init__(self, df, phase='train', transform=None):\n        self.df = df\n        self.transform = transform\n        self.phase = phase\n    \n    def __len__(self):\n        return len(self.df)\n\n    def __getitem__(self, idx):\n        x = np.zeros((IMG_SIZE[0], IMG_SIZE[1], IN_CHANS), dtype=np.uint8)\n        t = self.df.iloc[idx]\n        st_id = int(t['study_id'])\n        label = t[1:].values.astype(np.int64)\n        \n        # Sagittal T1\n        for i in range(0, 10, 1):\n            try:\n                p = f'./cvt_png/{st_id}/Sagittal T1/{i:03d}.png'\n                img = Image.open(p).convert('L')\n                img = np.array(img)\n                x[..., i] = img.astype(np.uint8)\n            except:\n                #print(f'failed to load on {st_id}, Sagittal T1')\n                pass\n            \n        # Sagittal T2/STIR\n        for i in range(0, 10, 1):\n            try:\n                p = f'./cvt_png/{st_id}/Sagittal T2_STIR/{i:03d}.png'\n                img = Image.open(p).convert('L')\n                img = np.array(img)\n                x[..., i+10] = img.astype(np.uint8)\n            except:\n                #print(f'failed to load on {st_id}, Sagittal T2/STIR')\n                pass\n            \n        # Axial T2\n        axt2 = glob(f'./cvt_png/{st_id}/Axial T2/*.png')\n        axt2 = sorted(axt2)\n    \n        step = len(axt2) / 10.0\n        st = len(axt2)/2.0 - 4.0*step\n        end = len(axt2)+0.0001\n                \n        for i, j in enumerate(np.arange(st, end, step)):\n            try:\n                p = axt2[max(0, int((j-0.5001).round()))]\n                img = Image.open(p).convert('L')\n                img = np.array(img)\n                x[..., i+20] = img.astype(np.uint8)\n            except:\n                #print(f'failed to load on {st_id}, Sagittal T2/STIR')\n                pass  \n            \n        assert np.sum(x)>0\n            \n        if self.transform is not None:\n            x = self.transform(image=x)['image']\n\n        x = x.transpose(2, 0, 1)\n                \n        return x, label","metadata":{"execution":{"iopub.status.busy":"2024-06-19T21:40:22.4265Z","iopub.execute_input":"2024-06-19T21:40:22.426875Z","iopub.status.idle":"2024-06-19T21:40:22.440558Z","shell.execute_reply.started":"2024-06-19T21:40:22.426843Z","shell.execute_reply":"2024-06-19T21:40:22.439812Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Define Data Augmentaion\nData augmentation is important because the number of images used for training is extremely small.\nSee [this notebook](https://www.kaggle.com/code/haqishen/1st-place-soluiton-code-small-ver) by [Qishen Ha](https://www.kaggle.com/haqishen) for help setting up this augmentation.","metadata":{}},{"cell_type":"code","source":"transforms_train = A.Compose([\n    A.RandomBrightnessContrast(brightness_limit=(-0.2, 0.2), contrast_limit=(-0.2, 0.2), p=AUG_PROB),\n    A.OneOf([\n        A.MotionBlur(blur_limit=5),\n        A.MedianBlur(blur_limit=5),\n        A.GaussianBlur(blur_limit=5),\n        A.GaussNoise(var_limit=(5.0, 30.0)),\n    ], p=AUG_PROB),\n\n    A.OneOf([\n        A.OpticalDistortion(distort_limit=1.0),\n        A.GridDistortion(num_steps=5, distort_limit=1.),\n        A.ElasticTransform(alpha=3),\n    ], p=AUG_PROB),\n\n    A.ShiftScaleRotate(shift_limit=0.1, scale_limit=0.1, rotate_limit=15, border_mode=0, p=AUG_PROB),\n    A.Resize(IMG_SIZE[0], IMG_SIZE[1]),\n    A.CoarseDropout(max_holes=16, max_height=64, max_width=64, min_holes=1, min_height=8, min_width=8, p=AUG_PROB),    \n    A.Normalize(mean=0.5, std=0.5)\n])\n\ntransforms_val = A.Compose([\n    A.Resize(IMG_SIZE[0], IMG_SIZE[1]),\n    A.Normalize(mean=0.5, std=0.5)\n])\n\nif not NOT_DEBUG or not AUG:\n    transforms_train = transforms_val","metadata":{"execution":{"iopub.status.busy":"2024-06-19T21:40:22.441585Z","iopub.execute_input":"2024-06-19T21:40:22.441895Z","iopub.status.idle":"2024-06-19T21:40:22.455303Z","shell.execute_reply.started":"2024-06-19T21:40:22.441873Z","shell.execute_reply":"2024-06-19T21:40:22.45448Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Trying Data Loader\nChecking if the data loader works properly.","metadata":{}},{"cell_type":"code","source":"tmp_ds = RSNA24Dataset(df, phase='train', transform=transforms_train)\ntmp_dl = DataLoader(\n            tmp_ds,\n            batch_size=1,\n            shuffle=False,\n            pin_memory=True,\n            drop_last=False,\n            num_workers=0\n            )\n\nfor i, (x, t) in enumerate(tmp_dl):\n    if i==2:break\n    print('x stat:', x.shape, x.min(), x.max(),x.mean(), x.std())\n    print(t, t.shape)\n    y = x.numpy().transpose(0,2,3,1)[0,...,:3]\n    y = (y + 1) / 2\n    plt.imshow(y)\n    plt.show()\n    print('y stat:', y.shape, y.min(), y.max(),y.mean(), y.std())\n    print()\nplt.close()\ndel tmp_ds, tmp_dl","metadata":{"execution":{"iopub.status.busy":"2024-06-19T21:40:22.456607Z","iopub.execute_input":"2024-06-19T21:40:22.457145Z","iopub.status.idle":"2024-06-19T21:40:24.052959Z","shell.execute_reply.started":"2024-06-19T21:40:22.457114Z","shell.execute_reply":"2024-06-19T21:40:24.051964Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Define Model\nWe use timm, which is commonly used for image classification.","metadata":{}},{"cell_type":"code","source":"class RSNA24Model(nn.Module):\n    def __init__(self, model_name, in_c=30, n_classes=75, pretrained=True, features_only=False):\n        super().__init__()\n        self.model = timm.create_model(\n                                    model_name,\n                                    pretrained=pretrained, \n                                    features_only=features_only,\n                                    in_chans=in_c,\n                                    num_classes=n_classes,\n                                    global_pool='avg'\n                                    )\n    \n    def forward(self, x):\n        y = self.model(x)\n        return y","metadata":{"execution":{"iopub.status.busy":"2024-06-19T21:40:24.056476Z","iopub.execute_input":"2024-06-19T21:40:24.056776Z","iopub.status.idle":"2024-06-19T21:40:24.062597Z","shell.execute_reply.started":"2024-06-19T21:40:24.05675Z","shell.execute_reply":"2024-06-19T21:40:24.061717Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Testing Model\nChecking if the model works properly.","metadata":{}},{"cell_type":"code","source":"m = RSNA24Model(MODEL_NAME, in_c=IN_CHANS, n_classes=N_CLASSES, pretrained=False)\ni = torch.randn(2, IN_CHANS, 512, 512)\nout = m(i)\nfor o in out:\n    print(o.shape, o.min(), o.max())","metadata":{"execution":{"iopub.status.busy":"2024-06-19T21:40:24.063837Z","iopub.execute_input":"2024-06-19T21:40:24.064611Z","iopub.status.idle":"2024-06-19T21:40:26.056873Z","shell.execute_reply.started":"2024-06-19T21:40:24.064579Z","shell.execute_reply":"2024-06-19T21:40:26.055909Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"del m, i, out","metadata":{"execution":{"iopub.status.busy":"2024-06-19T21:40:26.058093Z","iopub.execute_input":"2024-06-19T21:40:26.058399Z","iopub.status.idle":"2024-06-19T21:40:26.064153Z","shell.execute_reply.started":"2024-06-19T21:40:26.058373Z","shell.execute_reply":"2024-06-19T21:40:26.063316Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Train loop","metadata":{}},{"cell_type":"code","source":"%time\n#autocast = torch.cuda.amp.autocast(enabled=USE_AMP, dtype=torch.bfloat16) # if your gpu is newer Ampere, you can use this, lesser appearance of nan than half\nautocast = torch.cuda.amp.autocast(enabled=USE_AMP, dtype=torch.half) # you can use with T4 gpu. or newer\nscaler = torch.cuda.amp.GradScaler(enabled=USE_AMP, init_scale=4096)\n\nval_losses = []\ntrain_losses = []\nskf = KFold(n_splits=N_FOLDS, shuffle=True, random_state=SEED)\nfor fold, (trn_idx, val_idx) in enumerate(skf.split(range(len(df)))):\n    if NOT_DEBUG == False:\n        if fold == 1: break;\n    if fold not in SELECTED_FOLDS: \n        print(f\"Jump fold {fold}\")\n        continue;\n    else:\n        print('#'*30)\n        print(f'Start fold {fold}')\n        print('#'*30)\n        print(len(trn_idx), len(val_idx))\n        df_train = df.iloc[trn_idx]\n        df_valid = df.iloc[val_idx]\n\n        train_ds = RSNA24Dataset(df_train, phase='train', transform=transforms_train)\n        train_dl = DataLoader(\n                    train_ds,\n                    batch_size=BATCH_SIZE,\n                    shuffle=True,\n                    pin_memory=True,\n                    drop_last=True,\n                    num_workers=N_WORKERS\n                    )\n\n        valid_ds = RSNA24Dataset(df_valid, phase='valid', transform=transforms_val)\n        valid_dl = DataLoader(\n                    valid_ds,\n                    batch_size=BATCH_SIZE*2,\n                    shuffle=False,\n                    pin_memory=True,\n                    drop_last=False,\n                    num_workers=N_WORKERS\n                    )\n\n        model = RSNA24Model(MODEL_NAME, IN_CHANS, N_CLASSES, pretrained=True)\n        fname = f'{OUTPUT_DIR}/best_wll_model_fold-{fold}.pt'\n        if os.path.exists(fname):\n            model = RSNA24Model(MODEL_NAME, IN_CHANS, N_CLASSES, pretrained=False)\n            model.load_state_dict(torch.load(fname))\n        model.to(device)\n\n        optimizer = AdamW(model.parameters(), lr=LR, weight_decay=WD)\n\n        warmup_steps = EPOCHS/10 * len(train_dl) // GRAD_ACC\n        num_total_steps = EPOCHS * len(train_dl) // GRAD_ACC\n        num_cycles = 0.475\n        scheduler = get_cosine_schedule_with_warmup(optimizer,\n                                                    num_warmup_steps=warmup_steps,\n                                                    num_training_steps=num_total_steps,\n                                                    num_cycles=num_cycles)\n\n        weights = torch.tensor([1.0, 2.0, 4.0])\n        criterion = nn.CrossEntropyLoss(weight=weights.to(device))\n\n        best_loss = 1.2\n        es_step = 0\n\n        for epoch in range(1, EPOCHS+1):\n            print(f'start epoch {epoch}')\n            model.train()\n            total_loss = 0\n            with tqdm(train_dl, leave=True) as pbar:\n                optimizer.zero_grad()\n                for idx, (x, t) in enumerate(pbar):  \n                    x = x.to(device)\n                    t = t.to(device)\n\n                    with autocast:\n                        loss = 0\n                        y = model(x)\n                        for col in range(N_LABELS):\n                            pred = y[:,col*3:col*3+3]\n                            gt = t[:,col]\n                            loss = loss + criterion(pred, gt) / N_LABELS\n\n                        total_loss += loss.item()\n                        if GRAD_ACC > 1:\n                            loss = loss / GRAD_ACC\n\n                    if not math.isfinite(loss):\n                        print(f\"Loss is {loss}, stopping training\")\n                        sys.exit(1)\n\n                    pbar.set_postfix(\n                        OrderedDict(\n                            loss=f'{loss.item()*GRAD_ACC:.6f}',\n                            lr=f'{optimizer.param_groups[0][\"lr\"]:.3e}'\n                        )\n                    )\n                    scaler.scale(loss).backward()\n\n                    torch.nn.utils.clip_grad_norm_(model.parameters(), MAX_GRAD_NORM or 1e9)\n\n                    if (idx + 1) % GRAD_ACC == 0:\n                        scaler.step(optimizer)\n                        scaler.update()\n                        optimizer.zero_grad()\n                        if scheduler is not None:\n                            scheduler.step()                    \n\n            train_loss = total_loss/len(train_dl)\n            print(f'train_loss:{train_loss:.6f}')\n            train_losses.append(train_loss)\n            total_loss = 0\n\n            model.eval()\n            with tqdm(valid_dl, leave=True) as pbar:\n                with torch.no_grad():\n                    for idx, (x, t) in enumerate(pbar):\n\n                        x = x.to(device)\n                        t = t.to(device)\n\n                        with autocast:\n                            loss = 0\n                            loss_ema = 0\n                            y = model(x)\n                            for col in range(N_LABELS):\n                                pred = y[:,col*3:col*3+3]\n                                gt = t[:,col]\n\n                                loss = loss + criterion(pred, gt) / N_LABELS\n                                y_pred = pred.float()\n\n                            total_loss += loss.item()   \n\n            val_loss = total_loss/len(valid_dl)\n            print(f'val_loss:{val_loss:.6f}')\n            val_losses.append(val_loss)\n            if val_loss < best_loss:\n\n                if device!='cuda:0':\n                    model.to('cuda:0')                \n\n                print(f'epoch:{epoch}, best weighted_logloss updated from {best_loss:.6f} to {val_loss:.6f}')\n                best_loss = val_loss\n                fname = f'{OUTPUT_DIR}/best_wll_model_fold-{fold}.pt'\n                torch.save(model.state_dict(), fname)\n                print(f'{fname} is saved')\n                es_step = 0\n\n                if device!='cuda:0':\n                    model.to(device)\n\n            else:\n                es_step += 1\n                if es_step >= EARLY_STOPPING_EPOCH:\n                    print('early stopping')\n                    break  \n                                ","metadata":{"execution":{"iopub.status.busy":"2024-06-19T21:40:26.065609Z","iopub.execute_input":"2024-06-19T21:40:26.065876Z","iopub.status.idle":"2024-06-19T21:49:17.977872Z","shell.execute_reply.started":"2024-06-19T21:40:26.065853Z","shell.execute_reply":"2024-06-19T21:49:17.976718Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Loss Plot","metadata":{}},{"cell_type":"code","source":"## Plot Losses during training\nimport matplotlib.pyplot as plt\nplt.figure(figsize=(10, 5))\nplt.plot(train_losses, label='Training Loss', color='blue')\nplt.plot(val_losses, label='Validation Loss', color='red')\nplt.title('Training and Validation Loss')\nplt.xlabel('Epochs')\nplt.ylabel('Loss')\nplt.legend()\nplt.grid(True)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-06-19T21:49:17.982089Z","iopub.execute_input":"2024-06-19T21:49:17.982924Z","iopub.status.idle":"2024-06-19T21:49:18.311871Z","shell.execute_reply.started":"2024-06-19T21:49:17.98289Z","shell.execute_reply":"2024-06-19T21:49:18.310974Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Calculation CV","metadata":{}},{"cell_type":"code","source":"for dirname, _, filenames in os.walk(OUTPUT_DIR):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","metadata":{"execution":{"iopub.status.busy":"2024-06-19T21:49:18.313229Z","iopub.execute_input":"2024-06-19T21:49:18.313921Z","iopub.status.idle":"2024-06-19T21:49:18.319794Z","shell.execute_reply.started":"2024-06-19T21:49:18.313884Z","shell.execute_reply":"2024-06-19T21:49:18.318842Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"cv = 0\ny_preds = []\nlabels = []\nweights = torch.tensor([1.0, 2.0, 4.0])\ncriterion2 = nn.CrossEntropyLoss(weight=weights)\n\n## TODO: Modify EXIST_FOLDS by how many fold you've trained\nEXIST_FOLDS = [0,]\nfor fold, (trn_idx, val_idx) in enumerate(skf.split(range(len(df)))):\n#     if NOT_DEBUG == False:\n#         if fold == 1: break;\n    if fold not in EXIST_FOLDS: \n        print(f\"Jump fold {fold}\")\n        continue;\n    else:\n        print('#'*30)\n        print(f'Start fold {fold}')\n        print('#'*30)\n        df_valid = df.iloc[val_idx]\n        valid_ds = RSNA24Dataset(df_valid, phase='valid', transform=transforms_val)\n        valid_dl = DataLoader(\n                    valid_ds,\n                    batch_size=16,\n                    shuffle=False,\n                    pin_memory=True,\n                    drop_last=False,\n                    num_workers=N_WORKERS\n                    )\n\n        model = RSNA24Model(MODEL_NAME, IN_CHANS, N_CLASSES, pretrained=False)\n        # print(\"No internet read\")\n        fname = f'{OUTPUT_DIR}/best_wll_model_fold-{fold}.pt'\n        model.load_state_dict(torch.load(fname))\n        model.to(device)   \n\n        model.eval()\n        with tqdm(valid_dl, leave=True) as pbar:\n            with torch.no_grad():\n                for idx, (x, t) in enumerate(pbar):\n\n                    x = x.to(device)\n                    t = t.to(device)\n\n                    with autocast:\n                        y = model(x)\n                        for col in range(N_LABELS):\n                            pred = y[:,col*3:col*3+3]\n                            gt = t[:,col] \n                            y_pred = pred.float()\n                            y_preds.append(y_pred.cpu())\n                            labels.append(gt.cpu())\n\ny_preds = torch.cat(y_preds)\nlabels = torch.cat(labels)","metadata":{"execution":{"iopub.status.busy":"2024-06-19T21:49:18.321133Z","iopub.execute_input":"2024-06-19T21:49:18.321824Z","iopub.status.idle":"2024-06-19T21:50:53.047172Z","shell.execute_reply.started":"2024-06-19T21:49:18.32179Z","shell.execute_reply":"2024-06-19T21:50:53.045999Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"cv = criterion2(y_preds, labels)\nprint('cv score:', cv.item())","metadata":{"execution":{"iopub.status.busy":"2024-06-19T21:50:53.049354Z","iopub.execute_input":"2024-06-19T21:50:53.05024Z","iopub.status.idle":"2024-06-19T21:50:53.068868Z","shell.execute_reply.started":"2024-06-19T21:50:53.050197Z","shell.execute_reply":"2024-06-19T21:50:53.067979Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Calculation Competition Metrics\nThis will give a slightly different score, probably due to the different behavior for nan.","metadata":{}},{"cell_type":"code","source":"from sklearn.metrics import log_loss\ny_pred_np = y_preds.softmax(1).numpy()\nlabels_np = labels.numpy()\ny_pred_nan = np.zeros((y_preds.shape[0], 1))\ny_pred2 = np.concatenate([y_pred_nan, y_pred_np],axis=1)\nweights = []\nfor l in labels:\n    if l==0: weights.append(1)\n    elif l==1: weights.append(2)\n    elif l==2: weights.append(4)\n    else: weights.append(0)\ncv2 = log_loss(labels, y_pred2, normalize=True, sample_weight=weights)\nprint('cv score:', cv2)","metadata":{"execution":{"iopub.status.busy":"2024-06-19T21:50:53.070045Z","iopub.execute_input":"2024-06-19T21:50:53.070401Z","iopub.status.idle":"2024-06-19T21:50:53.311469Z","shell.execute_reply.started":"2024-06-19T21:50:53.070369Z","shell.execute_reply":"2024-06-19T21:50:53.310559Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"np.save(f'{OUTPUT_DIR}/labels.npy', labels_np)\nnp.save(f'{OUTPUT_DIR}/final_oof.npy', y_pred2)","metadata":{"execution":{"iopub.status.busy":"2024-06-19T21:50:53.313045Z","iopub.execute_input":"2024-06-19T21:50:53.313349Z","iopub.status.idle":"2024-06-19T21:50:53.319066Z","shell.execute_reply.started":"2024-06-19T21:50:53.313322Z","shell.execute_reply":"2024-06-19T21:50:53.31815Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# When predictions are random\ngets around 1.1.","metadata":{}},{"cell_type":"code","source":"random_pred = np.ones((y_preds.shape[0], 3)) / 3.0\ny_pred3 = np.concatenate([y_pred_nan, random_pred],axis=1)\ncv3 = log_loss(labels, y_pred3, normalize=True, sample_weight=weights)\nprint('random score:', cv3)","metadata":{"execution":{"iopub.status.busy":"2024-06-19T21:50:53.320237Z","iopub.execute_input":"2024-06-19T21:50:53.320614Z","iopub.status.idle":"2024-06-19T21:50:53.349645Z","shell.execute_reply.started":"2024-06-19T21:50:53.320579Z","shell.execute_reply":"2024-06-19T21:50:53.348677Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"!rm -r cvt_png","metadata":{"execution":{"iopub.status.busy":"2024-06-19T21:50:53.350795Z","iopub.execute_input":"2024-06-19T21:50:53.351082Z","iopub.status.idle":"2024-06-19T21:50:59.354632Z","shell.execute_reply.started":"2024-06-19T21:50:53.351057Z","shell.execute_reply":"2024-06-19T21:50:59.353239Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Continuing with the [Submission Baseline...](https://www.kaggle.com/code/itsuki9180/rsna2024-lsdc-submission-baseline)","metadata":{}},{"cell_type":"code","source":"","metadata":{},"outputs":[],"execution_count":null}]}