{
  "id": 475522,
  "title": "1st Place Solution (code updated)",
  "url": "/competitions/blood-vessel-segmentation/discussion/475522",
  "author_name": "Clevert",
  "post_date": "2024-02-08T18:44:11.031000",
  "votes": 100,
  "comment_count": 31,
  "views": 0,
  "content": "<p>First of all, we would like to thank Kaggle and the organizers for hosting such a great competition. And also thanks to <a href=\"https://www.kaggle.com/hengck23\" target=\"_blank\">@hengck23</a> for the amazing posts, <a href=\"https://www.kaggle.com/junkoda\" target=\"_blank\">@junkoda</a> for the metric implementation and all other participants for sharing their experiments.</p>\n<h1>Overview</h1>\n<p>Our final submission is an ensemble of two 2.5d convnext tiny unet with 3 channels, and the only differences between these two models are augmentation and number of epochs. Actually the best scored submission is not the selected ensemble but one single model of the ensemble which is 0.835 on private lb.</p>\n<h1>Data Preparation</h1>\n<p>We used all training data <strong>including</strong> kidney_1_voi.</p>\n<ul>\n<li>Multiview slice (x, y, z)</li>\n<li>Normalization: No normalization, just <code>image = image / 65535.0</code></li>\n<li>Whole slice instead of tiles and all slices resized or cropped to 1536x1536. </li>\n<li>Augmentations:</li>\n</ul>\n<pre><code>A.Compose([\n    A.HorizontalFlip(=0.5),\n    A.VerticalFlip(=0.5),\n    A.Transpose(=0.5),\n    A.Affine(scale={:(0.7, 1.3), :(0.7, 1.3)}, translate_percent={:(0, 0.1), :(0, 0.1)}, rotate=(-30, 30), shear=(-20, 20), =0.5),\n    A.RandomBrightnessContrast(=0.4, =0.4, =0.5),\n    A.OneOf([\n        A.Blur(=3, =0.2),\n        A.MedianBlur(=3, =0.2),\n    ], =1.0),\n    A.OneOf([\n        A.ElasticTransform(=1, =50, =10, =1, =0.5),\n        A.GridDistortion(=5, =0.1, =1, =0.5)\n    ], =0.4),\n    A.OneOf([\n        A.Resize(1536, 1536, cv2.INTER_LINEAR, =1),\n        A.Compose([\n            RandomResize(1536, 1536, =0.5, =0.5, =1),\n            A.PadIfNeeded(1536, 1536, =, =cv2.BORDER_REPLICATE, =1.0),\n            A.RandomCrop(1536, 1536, =1.0)\n        ], =1.0),\n    ], =1.0),\n    A.GaussNoise(=0.05, =0.2),\n])\n</code></pre>\n<ul>\n<li>Random 3D rotation to get slices that is not necessarily parallel to axes. The best scored submission used random 3d rotation for augmentation and trained for more epochs.<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4240322%2F07e6746d787af196eaceb6acf5d336fe%2F3drot.png?generation=1707409376653348&amp;alt=media\"></li>\n</ul>\n<h1>Modeling &amp; Training</h1>\n<ul>\n<li>We used unet from SMP with convnext tiny backbone, replaced BatchNorm and ReLU to GroupNorm and GELU and added an extra convolution stem. The input size for all models is 3x1536x1536.</li>\n</ul>\n<pre><code>self = nn(\n    nn(in_channels, out_channels, , , ),\n    (out_channels),\n)\n</code></pre>\n<ul>\n<li>For loss function, we used 1.0 focal loss, 1.0 dice loss, 0.01 <a href=\"https://arxiv.org/abs/1812.07032\" target=\"_blank\">boundary loss</a> and 1.0 custom loss. The custom loss is inspired by <a href=\"https://www.kaggle.com/hengck23\" target=\"_blank\">@hengck23</a>'s <a href=\"https://www.kaggle.com/competitions/blood-vessel-segmentation/discussion/456118#2583472\" target=\"_blank\">post</a> and <a href=\"https://www.kaggle.com/junkoda\" target=\"_blank\">@junkoda</a>'s <a href=\"https://www.kaggle.com/code/junkoda/fast-surface-dice-computation\" target=\"_blank\">metric implementation</a>.</li>\n</ul>\n<pre><code></code></pre>\n<ul>\n<li>For optimization, we used AdamW and CosineAnnealingLR from 1e-4 to 0 with warmup. All models were trained for 20 epochs with a batch size of 8 and 4 gradient accumulation steps, except for the model with 3d slice rotation augmentation which was trained for 30 epochs.</li>\n</ul>\n<h1>Inference</h1>\n<ul>\n<li>Inference on 3 axes with 8xTTA.</li>\n<li>We tried different resize methods for inference. For the best scored submission, all slices are simply resized to 3072x3072; for the selected submission, we used a dynamic scale factor that <code>(h*scale)*(w*scale)=3200*3200</code>.</li>\n<li>The threshold used for submission is 0.4, and the optimal threshold based on cv and lb is about 0.4~0.5.</li>\n<li><code>torch.compile()</code> gave about 2x acceleration so that we were able to inference with high resolution and TTAs.</li>\n</ul>\n<h1>What didn't work</h1>\n<ul>\n<li>3d models.</li>\n<li>External data and pseudo labels.</li>\n<li>Transformers.</li>\n<li>Stacking more slices (&gt;3) for 2.5d model.</li>\n</ul>\n<h1>Results</h1>\n<table>\n<thead>\n<tr>\n<th></th>\n<th><strong>Model</strong></th>\n<th><strong>Slice Rotation</strong></th>\n<th><strong>Inference size</strong></th>\n<th><strong>Public Score</strong></th>\n<th><strong>Private Score</strong></th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>1</td>\n<td>convnext_tiny</td>\n<td></td>\n<td>3072</td>\n<td>0.889</td>\n<td>0.682</td>\n</tr>\n<tr>\n<td>2</td>\n<td>convnext_tiny</td>\n<td>✓</td>\n<td>3072</td>\n<td>0.888</td>\n<td>0.830</td>\n</tr>\n<tr>\n<td>3</td>\n<td>convnext_tiny</td>\n<td>✓</td>\n<td>3072</td>\n<td>0.867</td>\n<td><strong>0.835</strong></td>\n</tr>\n<tr>\n<td>4</td>\n<td>ensemble(1+2)</td>\n<td>-</td>\n<td>3200</td>\n<td><strong>0.898</strong></td>\n<td>0.744(selected)</td>\n</tr>\n<tr>\n<td>5</td>\n<td>ensemble(1+2)</td>\n<td>-</td>\n<td>3200(dynamic)</td>\n<td>0.895</td>\n<td>0.774(selected)</td>\n</tr>\n</tbody>\n</table>\n<h1>Links</h1>\n<ul>\n<li><a href=\"https://github.com/jing1tian/blood-vessel-segmentation\" target=\"_blank\">training code</a></li>\n<li>inference code<ul>\n<li><a href=\"https://www.kaggle.com/code/clevert/sennet-1st-place-solution\" target=\"_blank\">final submission ensemble 0.774106</a></li>\n<li><a href=\"https://www.kaggle.com/code/clevert/sennet-unet-convnext-3d-rotation\" target=\"_blank\">3d rotate single model 0.835346</a></li></ul></li>\n</ul>",
  "messages": [
    {
      "id": 2643303,
      "postDate": "2024-02-08T18:44:11.033Z",
      "content": "<p>First of all, we would like to thank Kaggle and the organizers for hosting such a great competition. And also thanks to <a href=\"https://www.kaggle.com/hengck23\" target=\"_blank\">@hengck23</a> for the amazing posts, <a href=\"https://www.kaggle.com/junkoda\" target=\"_blank\">@junkoda</a> for the metric implementation and all other participants for sharing their experiments.</p>\n<h1>Overview</h1>\n<p>Our final submission is an ensemble of two 2.5d convnext tiny unet with 3 channels, and the only differences between these two models are augmentation and number of epochs. Actually the best scored submission is not the selected ensemble but one single model of the ensemble which is 0.835 on private lb.</p>\n<h1>Data Preparation</h1>\n<p>We used all training data <strong>including</strong> kidney_1_voi.</p>\n<ul>\n<li>Multiview slice (x, y, z)</li>\n<li>Normalization: No normalization, just <code>image = image / 65535.0</code></li>\n<li>Whole slice instead of tiles and all slices resized or cropped to 1536x1536. </li>\n<li>Augmentations:</li>\n</ul>\n<pre><code>A.Compose([\n    A.HorizontalFlip(=0.5),\n    A.VerticalFlip(=0.5),\n    A.Transpose(=0.5),\n    A.Affine(scale={:(0.7, 1.3), :(0.7, 1.3)}, translate_percent={:(0, 0.1), :(0, 0.1)}, rotate=(-30, 30), shear=(-20, 20), =0.5),\n    A.RandomBrightnessContrast(=0.4, =0.4, =0.5),\n    A.OneOf([\n        A.Blur(=3, =0.2),\n        A.MedianBlur(=3, =0.2),\n    ], =1.0),\n    A.OneOf([\n        A.ElasticTransform(=1, =50, =10, =1, =0.5),\n        A.GridDistortion(=5, =0.1, =1, =0.5)\n    ], =0.4),\n    A.OneOf([\n        A.Resize(1536, 1536, cv2.INTER_LINEAR, =1),\n        A.Compose([\n            RandomResize(1536, 1536, =0.5, =0.5, =1),\n            A.PadIfNeeded(1536, 1536, =, =cv2.BORDER_REPLICATE, =1.0),\n            A.RandomCrop(1536, 1536, =1.0)\n        ], =1.0),\n    ], =1.0),\n    A.GaussNoise(=0.05, =0.2),\n])\n</code></pre>\n<ul>\n<li>Random 3D rotation to get slices that is not necessarily parallel to axes. The best scored submission used random 3d rotation for augmentation and trained for more epochs.<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4240322%2F07e6746d787af196eaceb6acf5d336fe%2F3drot.png?generation=1707409376653348&amp;alt=media\"></li>\n</ul>\n<h1>Modeling &amp; Training</h1>\n<ul>\n<li>We used unet from SMP with convnext tiny backbone, replaced BatchNorm and ReLU to GroupNorm and GELU and added an extra convolution stem. The input size for all models is 3x1536x1536.</li>\n</ul>\n<pre><code>self = nn(\n    nn(in_channels, out_channels, , , ),\n    (out_channels),\n)\n</code></pre>\n<ul>\n<li>For loss function, we used 1.0 focal loss, 1.0 dice loss, 0.01 <a href=\"https://arxiv.org/abs/1812.07032\" target=\"_blank\">boundary loss</a> and 1.0 custom loss. The custom loss is inspired by <a href=\"https://www.kaggle.com/hengck23\" target=\"_blank\">@hengck23</a>'s <a href=\"https://www.kaggle.com/competitions/blood-vessel-segmentation/discussion/456118#2583472\" target=\"_blank\">post</a> and <a href=\"https://www.kaggle.com/junkoda\" target=\"_blank\">@junkoda</a>'s <a href=\"https://www.kaggle.com/code/junkoda/fast-surface-dice-computation\" target=\"_blank\">metric implementation</a>.</li>\n</ul>\n<pre><code></code></pre>\n<ul>\n<li>For optimization, we used AdamW and CosineAnnealingLR from 1e-4 to 0 with warmup. All models were trained for 20 epochs with a batch size of 8 and 4 gradient accumulation steps, except for the model with 3d slice rotation augmentation which was trained for 30 epochs.</li>\n</ul>\n<h1>Inference</h1>\n<ul>\n<li>Inference on 3 axes with 8xTTA.</li>\n<li>We tried different resize methods for inference. For the best scored submission, all slices are simply resized to 3072x3072; for the selected submission, we used a dynamic scale factor that <code>(h*scale)*(w*scale)=3200*3200</code>.</li>\n<li>The threshold used for submission is 0.4, and the optimal threshold based on cv and lb is about 0.4~0.5.</li>\n<li><code>torch.compile()</code> gave about 2x acceleration so that we were able to inference with high resolution and TTAs.</li>\n</ul>\n<h1>What didn't work</h1>\n<ul>\n<li>3d models.</li>\n<li>External data and pseudo labels.</li>\n<li>Transformers.</li>\n<li>Stacking more slices (&gt;3) for 2.5d model.</li>\n</ul>\n<h1>Results</h1>\n<table>\n<thead>\n<tr>\n<th></th>\n<th><strong>Model</strong></th>\n<th><strong>Slice Rotation</strong></th>\n<th><strong>Inference size</strong></th>\n<th><strong>Public Score</strong></th>\n<th><strong>Private Score</strong></th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>1</td>\n<td>convnext_tiny</td>\n<td></td>\n<td>3072</td>\n<td>0.889</td>\n<td>0.682</td>\n</tr>\n<tr>\n<td>2</td>\n<td>convnext_tiny</td>\n<td>✓</td>\n<td>3072</td>\n<td>0.888</td>\n<td>0.830</td>\n</tr>\n<tr>\n<td>3</td>\n<td>convnext_tiny</td>\n<td>✓</td>\n<td>3072</td>\n<td>0.867</td>\n<td><strong>0.835</strong></td>\n</tr>\n<tr>\n<td>4</td>\n<td>ensemble(1+2)</td>\n<td>-</td>\n<td>3200</td>\n<td><strong>0.898</strong></td>\n<td>0.744(selected)</td>\n</tr>\n<tr>\n<td>5</td>\n<td>ensemble(1+2)</td>\n<td>-</td>\n<td>3200(dynamic)</td>\n<td>0.895</td>\n<td>0.774(selected)</td>\n</tr>\n</tbody>\n</table>\n<h1>Links</h1>\n<ul>\n<li><a href=\"https://github.com/jing1tian/blood-vessel-segmentation\" target=\"_blank\">training code</a></li>\n<li>inference code<ul>\n<li><a href=\"https://www.kaggle.com/code/clevert/sennet-1st-place-solution\" target=\"_blank\">final submission ensemble 0.774106</a></li>\n<li><a href=\"https://www.kaggle.com/code/clevert/sennet-unet-convnext-3d-rotation\" target=\"_blank\">3d rotate single model 0.835346</a></li></ul></li>\n</ul>",
      "rawMarkdown": "First of all, we would like to thank Kaggle and the organizers for hosting such a great competition. And also thanks to @hengck23 for the amazing posts, @junkoda for the metric implementation and all other participants for sharing their experiments.\n# Overview\nOur final submission is an ensemble of two 2.5d convnext tiny unet with 3 channels, and the only differences between these two models are augmentation and number of epochs. Actually the best scored submission is not the selected ensemble but one single model of the ensemble which is 0.835 on private lb.\n\n# Data Preparation\nWe used all training data **including** kidney_1_voi.\n- Multiview slice (x, y, z)\n- Normalization: No normalization, just `image = image / 65535.0`\n- Whole slice instead of tiles and all slices resized or cropped to 1536x1536. \n- Augmentations:\n```\nA.Compose([\n    A.HorizontalFlip(p=0.5),\n    A.VerticalFlip(p=0.5),\n    A.Transpose(p=0.5),\n    A.Affine(scale={\"x\":(0.7, 1.3), \"y\":(0.7, 1.3)}, translate_percent={\"x\":(0, 0.1), \"y\":(0, 0.1)}, rotate=(-30, 30), shear=(-20, 20), p=0.5),\n    A.RandomBrightnessContrast(brightness_limit=0.4, contrast_limit=0.4, p=0.5),\n    A.OneOf([\n        A.Blur(blur_limit=3, p=0.2),\n        A.MedianBlur(blur_limit=3, p=0.2),\n    ], p=1.0),\n    A.OneOf([\n        A.ElasticTransform(alpha=1, sigma=50, alpha_affine=10, border_mode=1, p=0.5),\n        A.GridDistortion(num_steps=5, distort_limit=0.1, border_mode=1, p=0.5)\n    ], p=0.4),\n    A.OneOf([\n        A.Resize(1536, 1536, cv2.INTER_LINEAR, p=1),\n        A.Compose([\n            RandomResize(1536, 1536, scale_limit_x=0.5, scale_limit_y=0.5, p=1),\n            A.PadIfNeeded(1536, 1536, position=\"random\", border_mode=cv2.BORDER_REPLICATE, p=1.0),\n            A.RandomCrop(1536, 1536, p=1.0)\n        ], p=1.0),\n    ], p=1.0),\n    A.GaussNoise(var_limit=0.05, p=0.2),\n])\n```\n- Random 3D rotation to get slices that is not necessarily parallel to axes. The best scored submission used random 3d rotation for augmentation and trained for more epochs.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4240322%2F07e6746d787af196eaceb6acf5d336fe%2F3drot.png?generation=1707409376653348&alt=media)\n\n# Modeling & Training\n- We used unet from SMP with convnext tiny backbone, replaced BatchNorm and ReLU to GroupNorm and GELU and added an extra convolution stem. The input size for all models is 3x1536x1536.\n```\nself.extra_stem = nn.Sequential(\n    nn.Conv2d(in_channels, out_channels, 3, 2, 1),\n    LayerNorm2d(out_channels),\n)\n```\n- For loss function, we used 1.0 focal loss, 1.0 dice loss, 0.01 [boundary loss](https://arxiv.org/abs/1812.07032) and 1.0 custom loss. The custom loss is inspired by @hengck23's [post](https://www.kaggle.com/competitions/blood-vessel-segmentation/discussion/456118#2583472) and @junkoda's [metric implementation](https://www.kaggle.com/code/junkoda/fast-surface-dice-computation).\n```\nclass CustomLoss(nn.Module):\n    def __init__(self):\n        super().__init__()\n        power = 2**np.arange(0, 8).reshape(1, 1, 2, 2, 2).astype(np.float32)\n        area = create_table_neighbour_code_to_surface_area((1, 1, 1)).astype(np.float32)\n        self.power = nn.Parameter(torch.from_numpy(power), requires_grad=False)\n        self.kernel = nn.Parameter(torch.ones(1, 1, 2, 2, 2), requires_grad=False)\n        self.area = nn.Parameter(torch.from_numpy(area), requires_grad=False)\n        \n    def forward(self, preds, targets):\n        \"\"\"\n        preds: tensor of shape [bs, 1, d, h, w]\n        targets: tensor of shape [bs, 1, d, h, w]\n        \"\"\"\n        bsz = preds.shape[0]\n\n        # voxel logits to cube logits\n        foreground_probs = F.conv3d(F.logsigmoid(preds), self.kernel).exp().flatten(1)\n        background_probs = F.conv3d(F.logsigmoid(-preds), self.kernel).exp().flatten(1)\n        surface_probs = 1 - foreground_probs - background_probs\n\n        # ground truth\n        with torch.no_grad():\n            cubes_byte = F.conv3d(targets, self.power).to(torch.int32)\n            gt_area = self.area[cubes_byte.reshape(-1)].reshape(bsz, -1)\n            gt_foreground = (cubes_byte == 255).to(torch.float32).reshape(bsz, -1)\n            gt_background = (cubes_byte == 0).to(torch.float32).reshape(bsz, -1)\n            gt_surface = (gt_area > 0).to(torch.float32).reshape(bsz, -1)\n        \n        # dice\n        foreground_dice = 2 * (foreground_probs*gt_foreground).sum(-1) / (foreground_probs.sum(-1)+gt_foreground.sum(-1)).clamp(1e-6)\n        background_dice = 2 * (background_probs*gt_background).sum(-1) / (background_probs.sum(-1)+gt_background.sum(-1)).clamp(1e-6)\n        surface_dice = 2 * (surface_probs*gt_area).sum(-1) / ((surface_probs+gt_surface)*gt_area).sum(-1).clamp(1e-6)\n        dice = (foreground_dice + background_dice + surface_dice) / 3\n        return 1 - dice.mean()\n```\n- For optimization, we used AdamW and CosineAnnealingLR from 1e-4 to 0 with warmup. All models were trained for 20 epochs with a batch size of 8 and 4 gradient accumulation steps, except for the model with 3d slice rotation augmentation which was trained for 30 epochs.\n# Inference\n- Inference on 3 axes with 8xTTA.\n- We tried different resize methods for inference. For the best scored submission, all slices are simply resized to 3072x3072; for the selected submission, we used a dynamic scale factor that `(h*scale)*(w*scale)=3200*3200`.\n- The threshold used for submission is 0.4, and the optimal threshold based on cv and lb is about 0.4~0.5.\n- `torch.compile()` gave about 2x acceleration so that we were able to inference with high resolution and TTAs.\n# What didn't work\n- 3d models.\n- External data and pseudo labels.\n- Transformers.\n- Stacking more slices (>3) for 2.5d model.\n# Results\n|  | **Model** | **Slice Rotation** | **Inference size** | **Public Score**| **Private Score**|\n| --- | --- | --- | --- | --- | --- |\n| 1 | convnext_tiny |  | 3072 | 0.889 | 0.682 |\n| 2 | convnext_tiny | ✓ | 3072| 0.888 | 0.830 |\n| 3 | convnext_tiny | ✓ | 3072| 0.867 | **0.835** |\n| 4 | ensemble(1+2) | - | 3200 | **0.898** | 0.744(selected) |\n| 5 | ensemble(1+2) | - | 3200(dynamic) | 0.895 | 0.774(selected) |\n# Links\n- [training code](https://github.com/jing1tian/blood-vessel-segmentation)\n- inference code\n    - [final submission ensemble 0.774106](https://www.kaggle.com/code/clevert/sennet-1st-place-solution)\n    - [3d rotate single model 0.835346](https://www.kaggle.com/code/clevert/sennet-unet-convnext-3d-rotation)",
      "votes": 100
    },
    {
      "id": 2643591,
      "postDate": "2024-02-08T23:51:33.083Z",
      "content": "<p>thanks for the writeup. good work!</p>\n<p>As a reference, my implementation:</p>\n<ul>\n<li>convnext tiny</li>\n<li>random 3d slice</li>\n<li>boundary + interior bce loss weighing</li>\n<li>optimal local cv threshold and public: 0.4~0.5.</li>\n</ul>\n<hr>\n<p>comparing your results and mine, espically your very high private score, i think the cruical points are:</p>\n<ol>\n<li>training data including kidney_1_voi</li>\n<li>\"No normalization, just image = image / 65535.0\" + \"replaced BatchNorm and ReLU to GroupNorm\"<br>\n(this is local contrast, i.e. you are doing feature noramlisation instead of image intensity noramlisation)</li>\n<li>strong augmantation </li>\n<li>direct lb metric loss</li>\n</ol>",
      "rawMarkdown": "thanks for the writeup. good work!\n\nAs a reference, my implementation:\n-  convnext tiny\n-  random 3d slice\n- boundary + interior bce loss weighing\n- optimal local cv threshold and public: 0.4~0.5.\n\n---\n\ncomparing your results and mine, espically your very high private score, i think the cruical points are:\n1. training data including kidney_1_voi\n2. \"No normalization, just image = image / 65535.0\" + \"replaced BatchNorm and ReLU to GroupNorm\"\n(this is local contrast, i.e. you are doing feature noramlisation instead of image intensity noramlisation)\n3. strong augmantation \n4. direct lb metric loss",
      "votes": 10,
      "replies": [
        {
          "id": 2644167,
          "postDate": "2024-02-09T10:04:59.013Z",
          "content": "<p>nice work </p>",
          "rawMarkdown": "nice work "
        }
      ]
    },
    {
      "id": 2650997,
      "postDate": "2024-02-13T19:54:47.860Z",
      "content": "<p>congratulations on keeping the first place both on private <em>and</em> public leaderboards!! </p>\n<p>Can you tell me how do you introduce the <code>extra_stem</code> layer?</p>",
      "rawMarkdown": "congratulations on keeping the first place both on private *and* public leaderboards!! \n\nCan you tell me how do you introduce the `extra_stem` layer?",
      "votes": 3
    },
    {
      "id": 2643457,
      "postDate": "2024-02-08T20:55:56.230Z",
      "content": "<p>Cool! I see that 3d-rotation augmentation makes a big difference in the final score.</p>\n<p>What is the impact of the combination of the loss functions? <br>\nWhat score could be achieved with the same setup &amp; simple BCE or focal loss?</p>",
      "rawMarkdown": "Cool! I see that 3d-rotation augmentation makes a big difference in the final score.\n\nWhat is the impact of the combination of the loss functions? \nWhat score could be achieved with the same setup & simple BCE or focal loss?",
      "votes": 1,
      "replies": [
        {
          "id": 2644706,
          "postDate": "2024-02-09T16:20:54.253Z",
          "content": "<p>We started with focal + dice loss, so no results for simple BCE or focal loss. And we discovered boundary loss in early experiments which brought minimal improvement(&lt;1%) for cv and public lb. A combination of focal + dice + boundary loss with a single channel 2d model gave a score of 0.732/0.835 for private/public lb. As for the custom loss, it didn't bring any improvement for public/private score, but the 2.5d models with custom loss seem more stable for both public/private score.</p>",
          "rawMarkdown": "We started with focal + dice loss, so no results for simple BCE or focal loss. And we discovered boundary loss in early experiments which brought minimal improvement(<1%) for cv and public lb. A combination of focal + dice + boundary loss with a single channel 2d model gave a score of 0.732/0.835 for private/public lb. As for the custom loss, it didn't bring any improvement for public/private score, but the 2.5d models with custom loss seem more stable for both public/private score.",
          "votes": 1
        }
      ]
    },
    {
      "id": 2645553,
      "postDate": "2024-02-10T10:23:46.787Z",
      "content": "<p>Congratulations, Thank you for sharing the solution. It would be good for new learners if you provide complete notebook solution.</p>",
      "rawMarkdown": "Congratulations, Thank you for sharing the solution. It would be good for new learners if you provide complete notebook solution.",
      "votes": 2
    },
    {
      "id": 2643514,
      "postDate": "2024-02-08T21:52:38.837Z",
      "content": "<p>Congratulations, Thank you for sharing the solution. It would be good for new learners if you provide complete notebook solution. </p>",
      "rawMarkdown": "Congratulations, Thank you for sharing the solution. It would be good for new learners if you provide complete notebook solution. ",
      "votes": 2
    },
    {
      "id": 2643505,
      "postDate": "2024-02-08T21:37:27.077Z",
      "content": "<p>Very nice writeup and solution ! The attention to detail is very impressive ! Thank you for sharing.</p>\n<p>May I ask a few question:</p>\n<ul>\n<li>What's the reasoning behind the simple /  65535.0 norm ?</li>\n<li>Same question for Groupnorm and GeLU </li>\n<li>Is the aug scheme found throught iterative refinement or coming from another pipeline ?</li>\n</ul>\n<p>Thanks in advance for the answers. </p>\n<p>PS: The loss function is very impressive. I read <a href=\"https://www.kaggle.com/hengck23\" target=\"_blank\">@hengck23</a> discussion on it but didn't have the grit to actually implement it. Congrats for actually making it work.</p>",
      "rawMarkdown": "Very nice writeup and solution ! The attention to detail is very impressive ! Thank you for sharing.\n\nMay I ask a few question:\n- What's the reasoning behind the simple /  65535.0 norm ?\n- Same question for Groupnorm and GeLU \n- Is the aug scheme found throught iterative refinement or coming from another pipeline ?\n\nThanks in advance for the answers. \n\nPS: The loss function is very impressive. I read @hengck23 discussion on it but didn't have the grit to actually implement it. Congrats for actually making it work.",
      "votes": 2,
      "replies": [
        {
          "id": 2643943,
          "postDate": "2024-02-09T07:28:26.270Z",
          "content": "<p>\" didn't have the grit to actually implement it.\"</p>\n<p>Me too!</p>",
          "rawMarkdown": "\" didn't have the grit to actually implement it.\"\n\nMe too!"
        },
        {
          "id": 2644692,
          "postDate": "2024-02-09T16:14:35.133Z",
          "content": "<p>Thanks!</p>\n<ol>\n<li>We tried other normalization methods, but they did't work well for us.</li>\n<li>The GroupNorm is for gradient accumulation and the GELU is just an empirical choice.</li>\n<li>The augmentation scheme is found throught iterative refinement.</li>\n</ol>",
          "rawMarkdown": "Thanks!\n\n1. We tried other normalization methods, but they did't work well for us.\n2. The GroupNorm is for gradient accumulation and the GELU is just an empirical choice.\n3. The augmentation scheme is found throught iterative refinement.",
          "votes": 1
        }
      ]
    },
    {
      "id": 2795252,
      "postDate": "2024-05-05T17:52:29.327Z",
      "content": "<p>Are you planning to publish this? I would love to read the detailed methodology and ablation studies:)</p>",
      "rawMarkdown": "Are you planning to publish this? I would love to read the detailed methodology and ablation studies:)"
    },
    {
      "id": 2725958,
      "postDate": "2024-03-31T20:37:58.580Z",
      "content": "<p>Congraculations! and thank you for sharing the code</p>",
      "rawMarkdown": "Congraculations! and thank you for sharing the code"
    },
    {
      "id": 2662587,
      "postDate": "2024-02-22T02:26:32.170Z",
      "content": "<p>Congratulations! impressive work about norm.</p>",
      "rawMarkdown": "Congratulations! impressive work about norm."
    },
    {
      "id": 2654968,
      "postDate": "2024-02-16T15:45:41.787Z",
      "content": "<p>Congratulations!</p>",
      "rawMarkdown": "Congratulations!"
    },
    {
      "id": 2654958,
      "postDate": "2024-02-16T15:39:52.167Z",
      "content": "<p>Congratulations!!!</p>",
      "rawMarkdown": "Congratulations!!!"
    },
    {
      "id": 2654099,
      "postDate": "2024-02-15T21:08:15.727Z",
      "content": "<p>cogrates dear!!</p>",
      "rawMarkdown": "cogrates dear!!"
    },
    {
      "id": 2652112,
      "postDate": "2024-02-14T15:12:58.410Z",
      "content": "<p>Congratulations, Thank you for sharing the solution .</p>",
      "rawMarkdown": "Congratulations, Thank you for sharing the solution ."
    },
    {
      "id": 2645359,
      "postDate": "2024-02-10T07:16:46.503Z",
      "content": "<p>Congratulations on getting the first prize. Thanks for sharing the details of your solution. </p>",
      "rawMarkdown": "Congratulations on getting the first prize. Thanks for sharing the details of your solution. \n"
    },
    {
      "id": 2644726,
      "postDate": "2024-02-09T16:27:40.963Z",
      "content": "<p>Congratulations!</p>\n<p>Could you share how you handled CV vs LB scores? You don't talk about CV at all in your write up, does it mean that you only implemented new ideas and relied on LB? If not could you share the split and cv you had and did it correlate well with LB scores ?</p>",
      "rawMarkdown": "Congratulations!\n\nCould you share how you handled CV vs LB scores? You don't talk about CV at all in your write up, does it mean that you only implemented new ideas and relied on LB? If not could you share the split and cv you had and did it correlate well with LB scores ?",
      "replies": [
        {
          "id": 2645862,
          "postDate": "2024-02-10T14:11:44.537Z",
          "content": "<p>Thanks!<br>\nFor experimenting, we used kidney_3_dense for validation and other data for training. And we used IOU for local validation. For submission, we used all the data for training. The correlation between local cv and lb is not good. And because of the lack of data, our strategy was to improve lb score while ensuring the local cv improve or remain the same. Since we used all the data for training, we selected the last three checkpoints of one experiment for submissions, the lb score for different checkpoints varies a lot even with similar cv (the selection was based on lb). We also tried swa and ema, but didn't work well.<br>\nHere's some results for reference. The models for cv and lb were trained with same setting only that the model for submission was trained with all data.</p>\n<table>\n<thead>\n<tr>\n<th></th>\n<th>model</th>\n<th>epochs</th>\n<th>channels</th>\n<th>custom loss</th>\n<th>3d rotation</th>\n<th>IOU</th>\n<th>public score</th>\n<th>private score</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>1</td>\n<td>convnext tiny</td>\n<td>20</td>\n<td>1</td>\n<td></td>\n<td></td>\n<td>0.8116</td>\n<td>0.835</td>\n<td>0.732</td>\n</tr>\n<tr>\n<td>1</td>\n<td>convnext tiny</td>\n<td>20</td>\n<td>3</td>\n<td></td>\n<td></td>\n<td>0.8066</td>\n<td>0.852</td>\n<td>0.534</td>\n</tr>\n<tr>\n<td>2</td>\n<td>convnext tiny</td>\n<td>20</td>\n<td>3</td>\n<td>✓</td>\n<td></td>\n<td>0.8150</td>\n<td>0.889</td>\n<td>0.682</td>\n</tr>\n<tr>\n<td>3</td>\n<td>convnext tiny</td>\n<td>30</td>\n<td>3</td>\n<td>✓</td>\n<td>✓</td>\n<td>0.8153</td>\n<td>0.867</td>\n<td>0.835</td>\n</tr>\n<tr>\n<td>4</td>\n<td>convnext small</td>\n<td>30</td>\n<td>3</td>\n<td>✓</td>\n<td>✓</td>\n<td>0.8045</td>\n<td>0.848</td>\n<td>0.741</td>\n</tr>\n</tbody>\n</table>",
          "rawMarkdown": "Thanks!\nFor experimenting, we used kidney_3_dense for validation and other data for training. And we used IOU for local validation. For submission, we used all the data for training. The correlation between local cv and lb is not good. And because of the lack of data, our strategy was to improve lb score while ensuring the local cv improve or remain the same. Since we used all the data for training, we selected the last three checkpoints of one experiment for submissions, the lb score for different checkpoints varies a lot even with similar cv (the selection was based on lb). We also tried swa and ema, but didn't work well.\nHere's some results for reference. The models for cv and lb were trained with same setting only that the model for submission was trained with all data.\n\n|  | model | epochs | channels | custom loss | 3d rotation | IOU | public score | private score |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| 1 | convnext tiny | 20 | 1 |  |  | 0.8116 | 0.835 | 0.732 |\n| 1 | convnext tiny | 20 | 3 |  |  | 0.8066 | 0.852 | 0.534 |\n| 2 | convnext tiny | 20 | 3 | ✓ |  | 0.8150 | 0.889 | 0.682 |\n| 3 | convnext tiny | 30 | 3 | ✓ | ✓ | 0.8153 | 0.867 | 0.835 |\n| 4 | convnext small | 30 | 3 | ✓ | ✓ | 0.8045 | 0.848 | 0.741 |\n",
          "votes": 1
        }
      ]
    },
    {
      "id": 2644240,
      "postDate": "2024-02-09T10:43:19.180Z",
      "content": "<p>Well done and congratulations on staying on the top position both on public and private.</p>",
      "rawMarkdown": "Well done and congratulations on staying on the top position both on public and private.",
      "replies": [
        {
          "id": 2644652,
          "postDate": "2024-02-09T15:52:02.867Z",
          "content": "<p>Thanks!                           </p>",
          "rawMarkdown": "Thanks!                           "
        }
      ]
    },
    {
      "id": 2643663,
      "postDate": "2024-02-09T02:25:47.243Z",
      "content": "<p>Random 3D rotation and the loss are creative and insightful. Congratulations <a href=\"https://www.kaggle.com/clevert\" target=\"_blank\">@clevert</a> , this solution well deserves top1 👍. </p>\n<p>I would like to know how large the improvement brought by 1) Random 3D rotation, 2) the loss, 3) and the large inference size. May you provide more ablation study on these three key points?</p>",
      "rawMarkdown": "Random 3D rotation and the loss are creative and insightful. Congratulations @clevert , this solution well deserves top1 👍. \n\nI would like to know how large the improvement brought by 1) Random 3D rotation, 2) the loss, 3) and the large inference size. May you provide more ablation study on these three key points?",
      "replies": [
        {
          "id": 2643725,
          "postDate": "2024-02-09T03:55:02.043Z",
          "content": "<p>I would love to see this as well</p>",
          "rawMarkdown": "I would love to see this as well"
        },
        {
          "id": 2644663,
          "postDate": "2024-02-09T15:56:13.220Z",
          "content": "<p>Thank you!</p>\n<ol>\n<li>Random 3d rotation brought little improvement for both cv and public score. but for private score, the improvement is huge (0.682-&gt;0.835).</li>\n<li>The custom loss gave us ~2% boost for local cv, no improvement for public or private lb. But we noticed that the custom loss brought more stability and consistency for public and private scores.</li>\n<li>Large inference size is critical for the final score.  &gt;10% boost from 1536 to 3072 for both public and private scores.</li>\n</ol>",
          "rawMarkdown": "Thank you!\n\n1. Random 3d rotation brought little improvement for both cv and public score. but for private score, the improvement is huge (0.682->0.835).\n2. The custom loss gave us ~2% boost for local cv, no improvement for public or private lb. But we noticed that the custom loss brought more stability and consistency for public and private scores.\n3. Large inference size is critical for the final score.  >10% boost from 1536 to 3072 for both public and private scores.",
          "votes": 3,
          "replies": [
            {
              "id": 2644696,
              "postDate": "2024-02-09T16:15:50.083Z",
              "content": "<p><code>0.682-&gt;0.835</code> is a super huge improvements. Thank you for introducing this! 👍</p>",
              "rawMarkdown": "`0.682->0.835` is a super huge improvements. Thank you for introducing this! 👍"
            }
          ]
        }
      ]
    },
    {
      "id": 2643599,
      "postDate": "2024-02-09T00:19:21.083Z",
      "content": "<p>few question:</p>\n<p>first, why do you think ensemble(1+2)    performs worse in private? did you do post submission using better threshold adjustment? )maybe best threshold for ensmble is on 0.4?</p>\n<p>difference of 0.835 and 0.744/0.774 is large.  what is the cause? hence even at 0.835 private, results are still not stable (becuase of ground truth)?</p>\n<p>second, is there results for model using single channel (instead of 3 channel)? in this case you are not using marching cube loss.</p>\n<p>third, are you using convnext v1 or v2 (this include global response norm, grn)?</p>\n<hr>\n<p>\"We used all training data including kidney_1_voi.\" , this also means that sparsity is not important in annotation as ground truth?<br>\nhence the fine vessel are not detectable afterall?</p>",
      "rawMarkdown": "few question:\n\nfirst, why do you think ensemble(1+2)\tperforms worse in private? did you do post submission using better threshold adjustment? )maybe best threshold for ensmble is on 0.4?\n\ndifference of 0.835 and 0.744/0.774 is large.  what is the cause? hence even at 0.835 private, results are still not stable (becuase of ground truth)?\n\nsecond, is there results for model using single channel (instead of 3 channel)? in this case you are not using marching cube loss.\n\nthird, are you using convnext v1 or v2 (this include global response norm, grn)?\n\n\n----\n\n\"We used all training data including kidney_1_voi.\" , this also means that sparsity is not important in annotation as ground truth?\nhence the fine vessel are not detectable afterall?\n",
      "replies": [
        {
          "id": 2644679,
          "postDate": "2024-02-09T16:09:34.287Z",
          "content": "<p>1) We haven't done any post submission yet. Here's some existing results of different thresholds for ensemble. We will do more probes later. I think one potiential reason for the large difference might be the inconsistent spacing of the channel dimension for 2.5d models, while the inconsistency of other dimensions is alleviated by random scaling augmentation(3d resize might help?). And the 3d rotation might alleviate this problem by resampling? I realized this problem near the end the competition but didn't have time to dig into it.<br>\n2) Yes. We used single channel models in early experiments. Checking the submission history, the best single channel model scored 0.732/0.835 for private/public. But the single channel models are quite unstable and unreproducible, the scores ranging from 0.6 to 0.73 for private, 0.83~0.87 for public. For 2.5d models with custom loss, the private scores are about 0.67~0.69 for private and 0.86~0.88 for public.<br>\n3) We were using convnext v1 without grn.</p>\n<table>\n<thead>\n<tr>\n<th></th>\n<th>infer size</th>\n<th>threshold</th>\n<th>public score</th>\n<th>private score</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>1</td>\n<td>3072</td>\n<td>0.40</td>\n<td>0.897615</td>\n<td>0.751151</td>\n</tr>\n<tr>\n<td>2</td>\n<td>3072</td>\n<td>0.35</td>\n<td>0.895733</td>\n<td>0.767362</td>\n</tr>\n</tbody>\n</table>",
          "rawMarkdown": "\n1) We haven't done any post submission yet. Here's some existing results of different thresholds for ensemble. We will do more probes later. I think one potiential reason for the large difference might be the inconsistent spacing of the channel dimension for 2.5d models, while the inconsistency of other dimensions is alleviated by random scaling augmentation(3d resize might help?). And the 3d rotation might alleviate this problem by resampling? I realized this problem near the end the competition but didn't have time to dig into it.\n2) Yes. We used single channel models in early experiments. Checking the submission history, the best single channel model scored 0.732/0.835 for private/public. But the single channel models are quite unstable and unreproducible, the scores ranging from 0.6 to 0.73 for private, 0.83\\~0.87 for public. For 2.5d models with custom loss, the private scores are about 0.67\\~0.69 for private and 0.86\\~0.88 for public.\n3) We were using convnext v1 without grn.\n\n|  | infer size | threshold | public score | private score |\n| --- | --- | --- | --- | --- |\n| 1 | 3072 | 0.40 | 0.897615 | 0.751151 |\n| 2 | 3072 | 0.35 | 0.895733 | 0.767362 |",
          "votes": 2
        }
      ]
    },
    {
      "id": 2643584,
      "postDate": "2024-02-08T23:28:37.153Z",
      "content": "<p>Thank you for sharing! I express my deep respect that you won first place in both Public and Private. </p>\n<p>My question is, what data set configuration did you use to calculate the CV? (train and valid)</p>",
      "rawMarkdown": "Thank you for sharing! I express my deep respect that you won first place in both Public and Private. \n\nMy question is, what data set configuration did you use to calculate the CV? (train and valid)",
      "replies": [
        {
          "id": 2644688,
          "postDate": "2024-02-09T16:12:59.323Z",
          "content": "<p>For experimenting, we used kidney_3_dense for validation and other data for training. For submission, we used all the data for training.</p>",
          "rawMarkdown": "For experimenting, we used kidney_3_dense for validation and other data for training. For submission, we used all the data for training.",
          "votes": 2
        }
      ]
    },
    {
      "id": 3068245,
      "postDate": "2024-12-10T05:32:20.367Z",
      "rawMarkdown": "",
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 2643591,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "2024-02-08T23:51:33.083000",
      "content": "<p>thanks for the writeup. good work!</p>\n<p>As a reference, my implementation:</p>\n<ul>\n<li>convnext tiny</li>\n<li>random 3d slice</li>\n<li>boundary + interior bce loss weighing</li>\n<li>optimal local cv threshold and public: 0.4~0.5.</li>\n</ul>\n<hr>\n<p>comparing your results and mine, espically your very high private score, i think the cruical points are:</p>\n<ol>\n<li>training data including kidney_1_voi</li>\n<li>\"No normalization, just image = image / 65535.0\" + \"replaced BatchNorm and ReLU to GroupNorm\"<br>\n(this is local contrast, i.e. you are doing feature noramlisation instead of image intensity noramlisation)</li>\n<li>strong augmantation </li>\n<li>direct lb metric loss</li>\n</ol>",
      "votes": 10,
      "replies": [
        {
          "id": 2644167,
          "author_name": "Kavya_17",
          "author_url": "",
          "post_date": "2024-02-09T10:04:59.013000",
          "content": "<p>nice work </p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 2650997,
      "author_name": "Viktor Cikojevic",
      "author_url": "",
      "post_date": "2024-02-13T19:54:47.860000",
      "content": "<p>congratulations on keeping the first place both on private <em>and</em> public leaderboards!! </p>\n<p>Can you tell me how do you introduce the <code>extra_stem</code> layer?</p>",
      "votes": 3,
      "replies": []
    },
    {
      "id": 2643457,
      "author_name": "slime",
      "author_url": "",
      "post_date": "2024-02-08T20:55:56.230000",
      "content": "<p>Cool! I see that 3d-rotation augmentation makes a big difference in the final score.</p>\n<p>What is the impact of the combination of the loss functions? <br>\nWhat score could be achieved with the same setup &amp; simple BCE or focal loss?</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2644706,
          "author_name": "Clevert",
          "author_url": "",
          "post_date": "2024-02-09T16:20:54.253000",
          "content": "<p>We started with focal + dice loss, so no results for simple BCE or focal loss. And we discovered boundary loss in early experiments which brought minimal improvement(&lt;1%) for cv and public lb. A combination of focal + dice + boundary loss with a single channel 2d model gave a score of 0.732/0.835 for private/public lb. As for the custom loss, it didn't bring any improvement for public/private score, but the 2.5d models with custom loss seem more stable for both public/private score.</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 2645553,
      "author_name": "Tanishq dublish",
      "author_url": "",
      "post_date": "2024-02-10T10:23:46.787000",
      "content": "<p>Congratulations, Thank you for sharing the solution. It would be good for new learners if you provide complete notebook solution.</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 2643514,
      "author_name": "Huma Perveen",
      "author_url": "",
      "post_date": "2024-02-08T21:52:38.837000",
      "content": "<p>Congratulations, Thank you for sharing the solution. It would be good for new learners if you provide complete notebook solution. </p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 2643505,
      "author_name": "JEANMPIA",
      "author_url": "",
      "post_date": "2024-02-08T21:37:27.077000",
      "content": "<p>Very nice writeup and solution ! The attention to detail is very impressive ! Thank you for sharing.</p>\n<p>May I ask a few question:</p>\n<ul>\n<li>What's the reasoning behind the simple /  65535.0 norm ?</li>\n<li>Same question for Groupnorm and GeLU </li>\n<li>Is the aug scheme found throught iterative refinement or coming from another pipeline ?</li>\n</ul>\n<p>Thanks in advance for the answers. </p>\n<p>PS: The loss function is very impressive. I read <a href=\"https://www.kaggle.com/hengck23\" target=\"_blank\">@hengck23</a> discussion on it but didn't have the grit to actually implement it. Congrats for actually making it work.</p>",
      "votes": 2,
      "replies": [
        {
          "id": 2643943,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "2024-02-09T07:28:26.270000",
          "content": "<p>\" didn't have the grit to actually implement it.\"</p>\n<p>Me too!</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 2644692,
          "author_name": "Clevert",
          "author_url": "",
          "post_date": "2024-02-09T16:14:35.133000",
          "content": "<p>Thanks!</p>\n<ol>\n<li>We tried other normalization methods, but they did't work well for us.</li>\n<li>The GroupNorm is for gradient accumulation and the GELU is just an empirical choice.</li>\n<li>The augmentation scheme is found throught iterative refinement.</li>\n</ol>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 2795252,
      "author_name": "tzebin",
      "author_url": "",
      "post_date": "2024-05-05T17:52:29.327000",
      "content": "<p>Are you planning to publish this? I would love to read the detailed methodology and ablation studies:)</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 2725958,
      "author_name": "Metin Topyildiz",
      "author_url": "",
      "post_date": "2024-03-31T20:37:58.580000",
      "content": "<p>Congraculations! and thank you for sharing the code</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 2662587,
      "author_name": "jewe1ry",
      "author_url": "",
      "post_date": "2024-02-22T02:26:32.170000",
      "content": "<p>Congratulations! impressive work about norm.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 2654968,
      "author_name": "David Cano Rosillo",
      "author_url": "",
      "post_date": "2024-02-16T15:45:41.787000",
      "content": "<p>Congratulations!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 2654958,
      "author_name": "MarcoPG",
      "author_url": "",
      "post_date": "2024-02-16T15:39:52.167000",
      "content": "<p>Congratulations!!!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 2654099,
      "author_name": "malakkafaq",
      "author_url": "",
      "post_date": "2024-02-15T21:08:15.727000",
      "content": "<p>cogrates dear!!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 2652112,
      "author_name": "",
      "author_url": "",
      "post_date": "2024-02-14T15:12:58.410000",
      "content": "<p>Congratulations, Thank you for sharing the solution .</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 2645359,
      "author_name": "C R Suthikshn Kumar",
      "author_url": "",
      "post_date": "2024-02-10T07:16:46.503000",
      "content": "<p>Congratulations on getting the first prize. Thanks for sharing the details of your solution. </p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 2644726,
      "author_name": "Optimo",
      "author_url": "",
      "post_date": "2024-02-09T16:27:40.963000",
      "content": "<p>Congratulations!</p>\n<p>Could you share how you handled CV vs LB scores? You don't talk about CV at all in your write up, does it mean that you only implemented new ideas and relied on LB? If not could you share the split and cv you had and did it correlate well with LB scores ?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 2645862,
          "author_name": "Clevert",
          "author_url": "",
          "post_date": "2024-02-10T14:11:44.537000",
          "content": "<p>Thanks!<br>\nFor experimenting, we used kidney_3_dense for validation and other data for training. And we used IOU for local validation. For submission, we used all the data for training. The correlation between local cv and lb is not good. And because of the lack of data, our strategy was to improve lb score while ensuring the local cv improve or remain the same. Since we used all the data for training, we selected the last three checkpoints of one experiment for submissions, the lb score for different checkpoints varies a lot even with similar cv (the selection was based on lb). We also tried swa and ema, but didn't work well.<br>\nHere's some results for reference. The models for cv and lb were trained with same setting only that the model for submission was trained with all data.</p>\n<table>\n<thead>\n<tr>\n<th></th>\n<th>model</th>\n<th>epochs</th>\n<th>channels</th>\n<th>custom loss</th>\n<th>3d rotation</th>\n<th>IOU</th>\n<th>public score</th>\n<th>private score</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>1</td>\n<td>convnext tiny</td>\n<td>20</td>\n<td>1</td>\n<td></td>\n<td></td>\n<td>0.8116</td>\n<td>0.835</td>\n<td>0.732</td>\n</tr>\n<tr>\n<td>1</td>\n<td>convnext tiny</td>\n<td>20</td>\n<td>3</td>\n<td></td>\n<td></td>\n<td>0.8066</td>\n<td>0.852</td>\n<td>0.534</td>\n</tr>\n<tr>\n<td>2</td>\n<td>convnext tiny</td>\n<td>20</td>\n<td>3</td>\n<td>✓</td>\n<td></td>\n<td>0.8150</td>\n<td>0.889</td>\n<td>0.682</td>\n</tr>\n<tr>\n<td>3</td>\n<td>convnext tiny</td>\n<td>30</td>\n<td>3</td>\n<td>✓</td>\n<td>✓</td>\n<td>0.8153</td>\n<td>0.867</td>\n<td>0.835</td>\n</tr>\n<tr>\n<td>4</td>\n<td>convnext small</td>\n<td>30</td>\n<td>3</td>\n<td>✓</td>\n<td>✓</td>\n<td>0.8045</td>\n<td>0.848</td>\n<td>0.741</td>\n</tr>\n</tbody>\n</table>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 2644240,
      "author_name": "Yassine Alouini",
      "author_url": "",
      "post_date": "2024-02-09T10:43:19.180000",
      "content": "<p>Well done and congratulations on staying on the top position both on public and private.</p>",
      "votes": 0,
      "replies": [
        {
          "id": 2644652,
          "author_name": "Clevert",
          "author_url": "",
          "post_date": "2024-02-09T15:52:02.867000",
          "content": "<p>Thanks!                           </p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 2643663,
      "author_name": "ForcewithMe",
      "author_url": "",
      "post_date": "2024-02-09T02:25:47.243000",
      "content": "<p>Random 3D rotation and the loss are creative and insightful. Congratulations <a href=\"https://www.kaggle.com/clevert\" target=\"_blank\">@clevert</a> , this solution well deserves top1 👍. </p>\n<p>I would like to know how large the improvement brought by 1) Random 3D rotation, 2) the loss, 3) and the large inference size. May you provide more ablation study on these three key points?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 2643725,
          "author_name": "chemdatafarmer",
          "author_url": "",
          "post_date": "2024-02-09T03:55:02.043000",
          "content": "<p>I would love to see this as well</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 2644663,
          "author_name": "Clevert",
          "author_url": "",
          "post_date": "2024-02-09T15:56:13.220000",
          "content": "<p>Thank you!</p>\n<ol>\n<li>Random 3d rotation brought little improvement for both cv and public score. but for private score, the improvement is huge (0.682-&gt;0.835).</li>\n<li>The custom loss gave us ~2% boost for local cv, no improvement for public or private lb. But we noticed that the custom loss brought more stability and consistency for public and private scores.</li>\n<li>Large inference size is critical for the final score.  &gt;10% boost from 1536 to 3072 for both public and private scores.</li>\n</ol>",
          "votes": 3,
          "replies": [
            {
              "id": 2644696,
              "author_name": "ForcewithMe",
              "author_url": "",
              "post_date": "2024-02-09T16:15:50.083000",
              "content": "<p><code>0.682-&gt;0.835</code> is a super huge improvements. Thank you for introducing this! 👍</p>",
              "votes": 0,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 2643599,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "2024-02-09T00:19:21.083000",
      "content": "<p>few question:</p>\n<p>first, why do you think ensemble(1+2)    performs worse in private? did you do post submission using better threshold adjustment? )maybe best threshold for ensmble is on 0.4?</p>\n<p>difference of 0.835 and 0.744/0.774 is large.  what is the cause? hence even at 0.835 private, results are still not stable (becuase of ground truth)?</p>\n<p>second, is there results for model using single channel (instead of 3 channel)? in this case you are not using marching cube loss.</p>\n<p>third, are you using convnext v1 or v2 (this include global response norm, grn)?</p>\n<hr>\n<p>\"We used all training data including kidney_1_voi.\" , this also means that sparsity is not important in annotation as ground truth?<br>\nhence the fine vessel are not detectable afterall?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 2644679,
          "author_name": "Clevert",
          "author_url": "",
          "post_date": "2024-02-09T16:09:34.287000",
          "content": "<p>1) We haven't done any post submission yet. Here's some existing results of different thresholds for ensemble. We will do more probes later. I think one potiential reason for the large difference might be the inconsistent spacing of the channel dimension for 2.5d models, while the inconsistency of other dimensions is alleviated by random scaling augmentation(3d resize might help?). And the 3d rotation might alleviate this problem by resampling? I realized this problem near the end the competition but didn't have time to dig into it.<br>\n2) Yes. We used single channel models in early experiments. Checking the submission history, the best single channel model scored 0.732/0.835 for private/public. But the single channel models are quite unstable and unreproducible, the scores ranging from 0.6 to 0.73 for private, 0.83~0.87 for public. For 2.5d models with custom loss, the private scores are about 0.67~0.69 for private and 0.86~0.88 for public.<br>\n3) We were using convnext v1 without grn.</p>\n<table>\n<thead>\n<tr>\n<th></th>\n<th>infer size</th>\n<th>threshold</th>\n<th>public score</th>\n<th>private score</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>1</td>\n<td>3072</td>\n<td>0.40</td>\n<td>0.897615</td>\n<td>0.751151</td>\n</tr>\n<tr>\n<td>2</td>\n<td>3072</td>\n<td>0.35</td>\n<td>0.895733</td>\n<td>0.767362</td>\n</tr>\n</tbody>\n</table>",
          "votes": 2,
          "replies": []
        }
      ]
    },
    {
      "id": 2643584,
      "author_name": "Chikuwabu",
      "author_url": "",
      "post_date": "2024-02-08T23:28:37.153000",
      "content": "<p>Thank you for sharing! I express my deep respect that you won first place in both Public and Private. </p>\n<p>My question is, what data set configuration did you use to calculate the CV? (train and valid)</p>",
      "votes": 0,
      "replies": [
        {
          "id": 2644688,
          "author_name": "Clevert",
          "author_url": "",
          "post_date": "2024-02-09T16:12:59.323000",
          "content": "<p>For experimenting, we used kidney_3_dense for validation and other data for training. For submission, we used all the data for training.</p>",
          "votes": 2,
          "replies": []
        }
      ]
    },
    {
      "id": 3068245,
      "author_name": "",
      "author_url": "",
      "post_date": "2024-12-10T05:32:20.367000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2643303": "First of all, we would like to thank Kaggle and the organizers for hosting such a great competition. And also thanks to @hengck23 for the amazing posts, @junkoda for the metric implementation and all other participants for sharing their experiments.\n# Overview\nOur final submission is an ensemble of two 2.5d convnext tiny unet with 3 channels, and the only differences between these two models are augmentation and number of epochs. Actually the best scored submission is not the selected ensemble but one single model of the ensemble which is 0.835 on private lb.\n\n# Data Preparation\nWe used all training data **including** kidney_1_voi.\n- Multiview slice (x, y, z)\n- Normalization: No normalization, just `image = image / 65535.0`\n- Whole slice instead of tiles and all slices resized or cropped to 1536x1536. \n- Augmentations:\n```\nA.Compose([\n    A.HorizontalFlip(p=0.5),\n    A.VerticalFlip(p=0.5),\n    A.Transpose(p=0.5),\n    A.Affine(scale={\"x\":(0.7, 1.3), \"y\":(0.7, 1.3)}, translate_percent={\"x\":(0, 0.1), \"y\":(0, 0.1)}, rotate=(-30, 30), shear=(-20, 20), p=0.5),\n    A.RandomBrightnessContrast(brightness_limit=0.4, contrast_limit=0.4, p=0.5),\n    A.OneOf([\n        A.Blur(blur_limit=3, p=0.2),\n        A.MedianBlur(blur_limit=3, p=0.2),\n    ], p=1.0),\n    A.OneOf([\n        A.ElasticTransform(alpha=1, sigma=50, alpha_affine=10, border_mode=1, p=0.5),\n        A.GridDistortion(num_steps=5, distort_limit=0.1, border_mode=1, p=0.5)\n    ], p=0.4),\n    A.OneOf([\n        A.Resize(1536, 1536, cv2.INTER_LINEAR, p=1),\n        A.Compose([\n            RandomResize(1536, 1536, scale_limit_x=0.5, scale_limit_y=0.5, p=1),\n            A.PadIfNeeded(1536, 1536, position=\"random\", border_mode=cv2.BORDER_REPLICATE, p=1.0),\n            A.RandomCrop(1536, 1536, p=1.0)\n        ], p=1.0),\n    ], p=1.0),\n    A.GaussNoise(var_limit=0.05, p=0.2),\n])\n```\n- Random 3D rotation to get slices that is not necessarily parallel to axes. The best scored submission used random 3d rotation for augmentation and trained for more epochs.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4240322%2F07e6746d787af196eaceb6acf5d336fe%2F3drot.png?generation=1707409376653348&alt=media)\n\n# Modeling & Training\n- We used unet from SMP with convnext tiny backbone, replaced BatchNorm and ReLU to GroupNorm and GELU and added an extra convolution stem. The input size for all models is 3x1536x1536.\n```\nself.extra_stem = nn.Sequential(\n    nn.Conv2d(in_channels, out_channels, 3, 2, 1),\n    LayerNorm2d(out_channels),\n)\n```\n- For loss function, we used 1.0 focal loss, 1.0 dice loss, 0.01 [boundary loss](https://arxiv.org/abs/1812.07032) and 1.0 custom loss. The custom loss is inspired by @hengck23's [post](https://www.kaggle.com/competitions/blood-vessel-segmentation/discussion/456118#2583472) and @junkoda's [metric implementation](https://www.kaggle.com/code/junkoda/fast-surface-dice-computation).\n```\nclass CustomLoss(nn.Module):\n    def __init__(self):\n        super().__init__()\n        power = 2**np.arange(0, 8).reshape(1, 1, 2, 2, 2).astype(np.float32)\n        area = create_table_neighbour_code_to_surface_area((1, 1, 1)).astype(np.float32)\n        self.power = nn.Parameter(torch.from_numpy(power), requires_grad=False)\n        self.kernel = nn.Parameter(torch.ones(1, 1, 2, 2, 2), requires_grad=False)\n        self.area = nn.Parameter(torch.from_numpy(area), requires_grad=False)\n        \n    def forward(self, preds, targets):\n        \"\"\"\n        preds: tensor of shape [bs, 1, d, h, w]\n        targets: tensor of shape [bs, 1, d, h, w]\n        \"\"\"\n        bsz = preds.shape[0]\n\n        # voxel logits to cube logits\n        foreground_probs = F.conv3d(F.logsigmoid(preds), self.kernel).exp().flatten(1)\n        background_probs = F.conv3d(F.logsigmoid(-preds), self.kernel).exp().flatten(1)\n        surface_probs = 1 - foreground_probs - background_probs\n\n        # ground truth\n        with torch.no_grad():\n            cubes_byte = F.conv3d(targets, self.power).to(torch.int32)\n            gt_area = self.area[cubes_byte.reshape(-1)].reshape(bsz, -1)\n            gt_foreground = (cubes_byte == 255).to(torch.float32).reshape(bsz, -1)\n            gt_background = (cubes_byte == 0).to(torch.float32).reshape(bsz, -1)\n            gt_surface = (gt_area > 0).to(torch.float32).reshape(bsz, -1)\n        \n        # dice\n        foreground_dice = 2 * (foreground_probs*gt_foreground).sum(-1) / (foreground_probs.sum(-1)+gt_foreground.sum(-1)).clamp(1e-6)\n        background_dice = 2 * (background_probs*gt_background).sum(-1) / (background_probs.sum(-1)+gt_background.sum(-1)).clamp(1e-6)\n        surface_dice = 2 * (surface_probs*gt_area).sum(-1) / ((surface_probs+gt_surface)*gt_area).sum(-1).clamp(1e-6)\n        dice = (foreground_dice + background_dice + surface_dice) / 3\n        return 1 - dice.mean()\n```\n- For optimization, we used AdamW and CosineAnnealingLR from 1e-4 to 0 with warmup. All models were trained for 20 epochs with a batch size of 8 and 4 gradient accumulation steps, except for the model with 3d slice rotation augmentation which was trained for 30 epochs.\n# Inference\n- Inference on 3 axes with 8xTTA.\n- We tried different resize methods for inference. For the best scored submission, all slices are simply resized to 3072x3072; for the selected submission, we used a dynamic scale factor that `(h*scale)*(w*scale)=3200*3200`.\n- The threshold used for submission is 0.4, and the optimal threshold based on cv and lb is about 0.4~0.5.\n- `torch.compile()` gave about 2x acceleration so that we were able to inference with high resolution and TTAs.\n# What didn't work\n- 3d models.\n- External data and pseudo labels.\n- Transformers.\n- Stacking more slices (>3) for 2.5d model.\n# Results\n|  | **Model** | **Slice Rotation** | **Inference size** | **Public Score**| **Private Score**|\n| --- | --- | --- | --- | --- | --- |\n| 1 | convnext_tiny |  | 3072 | 0.889 | 0.682 |\n| 2 | convnext_tiny | ✓ | 3072| 0.888 | 0.830 |\n| 3 | convnext_tiny | ✓ | 3072| 0.867 | **0.835** |\n| 4 | ensemble(1+2) | - | 3200 | **0.898** | 0.744(selected) |\n| 5 | ensemble(1+2) | - | 3200(dynamic) | 0.895 | 0.774(selected) |\n# Links\n- [training code](https://github.com/jing1tian/blood-vessel-segmentation)\n- inference code\n    - [final submission ensemble 0.774106](https://www.kaggle.com/code/clevert/sennet-1st-place-solution)\n    - [3d rotate single model 0.835346](https://www.kaggle.com/code/clevert/sennet-unet-convnext-3d-rotation)",
    "2643591": "thanks for the writeup. good work!\n\nAs a reference, my implementation:\n-  convnext tiny\n-  random 3d slice\n- boundary + interior bce loss weighing\n- optimal local cv threshold and public: 0.4~0.5.\n\n---\n\ncomparing your results and mine, espically your very high private score, i think the cruical points are:\n1. training data including kidney_1_voi\n2. \"No normalization, just image = image / 65535.0\" + \"replaced BatchNorm and ReLU to GroupNorm\"\n(this is local contrast, i.e. you are doing feature noramlisation instead of image intensity noramlisation)\n3. strong augmantation \n4. direct lb metric loss",
    "2650997": "congratulations on keeping the first place both on private *and* public leaderboards!! \n\nCan you tell me how do you introduce the `extra_stem` layer?",
    "2643457": "Cool! I see that 3d-rotation augmentation makes a big difference in the final score.\n\nWhat is the impact of the combination of the loss functions? \nWhat score could be achieved with the same setup & simple BCE or focal loss?",
    "2645553": "Congratulations, Thank you for sharing the solution. It would be good for new learners if you provide complete notebook solution.",
    "2643514": "Congratulations, Thank you for sharing the solution. It would be good for new learners if you provide complete notebook solution. ",
    "2643505": "Very nice writeup and solution ! The attention to detail is very impressive ! Thank you for sharing.\n\nMay I ask a few question:\n- What's the reasoning behind the simple /  65535.0 norm ?\n- Same question for Groupnorm and GeLU \n- Is the aug scheme found throught iterative refinement or coming from another pipeline ?\n\nThanks in advance for the answers. \n\nPS: The loss function is very impressive. I read @hengck23 discussion on it but didn't have the grit to actually implement it. Congrats for actually making it work.",
    "2795252": "Are you planning to publish this? I would love to read the detailed methodology and ablation studies:)",
    "2725958": "Congraculations! and thank you for sharing the code",
    "2662587": "Congratulations! impressive work about norm.",
    "2654968": "Congratulations!",
    "2654958": "Congratulations!!!",
    "2654099": "cogrates dear!!",
    "2652112": "Congratulations, Thank you for sharing the solution .",
    "2645359": "Congratulations on getting the first prize. Thanks for sharing the details of your solution. \n",
    "2644726": "Congratulations!\n\nCould you share how you handled CV vs LB scores? You don't talk about CV at all in your write up, does it mean that you only implemented new ideas and relied on LB? If not could you share the split and cv you had and did it correlate well with LB scores ?",
    "2644240": "Well done and congratulations on staying on the top position both on public and private.",
    "2643663": "Random 3D rotation and the loss are creative and insightful. Congratulations @clevert , this solution well deserves top1 👍. \n\nI would like to know how large the improvement brought by 1) Random 3D rotation, 2) the loss, 3) and the large inference size. May you provide more ablation study on these three key points?",
    "2643599": "few question:\n\nfirst, why do you think ensemble(1+2)\tperforms worse in private? did you do post submission using better threshold adjustment? )maybe best threshold for ensmble is on 0.4?\n\ndifference of 0.835 and 0.744/0.774 is large.  what is the cause? hence even at 0.835 private, results are still not stable (becuase of ground truth)?\n\nsecond, is there results for model using single channel (instead of 3 channel)? in this case you are not using marching cube loss.\n\nthird, are you using convnext v1 or v2 (this include global response norm, grn)?\n\n\n----\n\n\"We used all training data including kidney_1_voi.\" , this also means that sparsity is not important in annotation as ground truth?\nhence the fine vessel are not detectable afterall?\n",
    "2643584": "Thank you for sharing! I express my deep respect that you won first place in both Public and Private. \n\nMy question is, what data set configuration did you use to calculate the CV? (train and valid)",
    "3068245": ""
  }
}