Thank you for host, I learned a lot from this competiton!
\nalso special thanks to @hengck23 . I refer @hengck23 discussion topics many times.
(nn.Module):\n ():\n (SenUNetStem, self).__init__()\n kwargs = (\n in_chans=in_chans,\n features_only=,\n \n pretrained=,\n out_indices=((encoder_depth)),\n )\n self.conv_stem = Conv2dReLU(in_chans, , , use_layernorm=, padding=)\n self.encoder = timm.create_model(encoder_name, **kwargs)\n self._out_channels = [\n ,\n ] + self.encoder.feature_info.channels()\n\n self.decoder = UnetDecoder(\n encoder_channels=self._out_channels,\n decoder_channels=decoder_channels,\n n_blocks=encoder_depth,\n use_batchnorm=decoder_use_batchnorm,\n center= encoder_name.startswith() ,\n attention_type=decoder_attention_type,\n )\n\n self.segmentation_head = SegmentationHead(\n in_channels=decoder_channels[-] + ,\n out_channels=classes,\n activation=activation,\n kernel_size=,\n )\n\n self.n_time = n_time\n self.pickup_index = pickup_index\n\n ():\n B, C, H, W = x.shape\n h = (H//)*\n w = (W//)*\n x = x[:,:,:h,:w]\n stem = self.conv_stem(x)\n features = self.encoder(x) \n features = [\n stem,\n ] + features\n\n decoder_output = self.decoder(*features)\n masks = self.segmentation_head(decoder_output)\n masks = F.pad(masks,[,W-w,,H-h,,,,], mode=, value=)\n\n masks[:,]\n\nSenUNetStem(\n encoder_name=,\n classes=,\n activation=,\n )\nI focus on regularization trick.
\nbecause this competiton have unstable cv, public is not related.
\nMoreover host shared public/Private LB information images, I hink it indicate unstable.
train_aug = A.Compose([\n A.HorizontalFlip(=0.5),\n A.VerticalFlip(=0.5),\n A.RandomBrightness(=0.1, =0.7),\n A.OneOf([\n A.GaussNoise(var_limit=[10, 50]),\n A.GaussianBlur(),\n A.MotionBlur(),\n A.MedianBlur(=3),\n ], =0.4),\n A.OneOf([\n A.GridDistortion(=5, =0.3, =1.0),\n A.OpticalDistortion(=1., =1.0)\n ],=0.2),\n A.ShiftScaleRotate(=0.7, =0.5, =0.2, =30),\n A.CoarseDropout(=1, =0.25, =0.25),\n ToTensorV2(=)\n ])\nInference is xy, xz, yz axis, and crop 512, stride 256.
\nProbability threshold(=sigmoid output). I didn't use percentile which method used many public notebook and past segmentation competiton(e.g. Volcano).
\nBecause I checked percentile threshold in local cv, it's not stable. I didn't use it.
https://www.kaggle.com/code/tereka/simpleunet-xy-xz-yz-v2-nbp-b749ff/notebook
", "messages": [ { "id": "2640615", "postDate": "02/07/2024 02:41:30", "content": "Thank you for host, I learned a lot from this competiton!
\nalso special thanks to @hengck23 . I refer @hengck23 discussion topics many times.
(nn.Module):\n ():\n (SenUNetStem, self).__init__()\n kwargs = (\n in_chans=in_chans,\n features_only=,\n \n pretrained=,\n out_indices=((encoder_depth)),\n )\n self.conv_stem = Conv2dReLU(in_chans, , , use_layernorm=, padding=)\n self.encoder = timm.create_model(encoder_name, **kwargs)\n self._out_channels = [\n ,\n ] + self.encoder.feature_info.channels()\n\n self.decoder = UnetDecoder(\n encoder_channels=self._out_channels,\n decoder_channels=decoder_channels,\n n_blocks=encoder_depth,\n use_batchnorm=decoder_use_batchnorm,\n center= encoder_name.startswith() ,\n attention_type=decoder_attention_type,\n )\n\n self.segmentation_head = SegmentationHead(\n in_channels=decoder_channels[-] + ,\n out_channels=classes,\n activation=activation,\n kernel_size=,\n )\n\n self.n_time = n_time\n self.pickup_index = pickup_index\n\n ():\n B, C, H, W = x.shape\n h = (H//)*\n w = (W//)*\n x = x[:,:,:h,:w]\n stem = self.conv_stem(x)\n features = self.encoder(x) \n features = [\n stem,\n ] + features\n\n decoder_output = self.decoder(*features)\n masks = self.segmentation_head(decoder_output)\n masks = F.pad(masks,[,W-w,,H-h,,,,], mode=, value=)\n\n masks[:,]\n\nSenUNetStem(\n encoder_name=,\n classes=,\n activation=,\n )\nI focus on regularization trick.
\nbecause this competiton have unstable cv, public is not related.
\nMoreover host shared public/Private LB information images, I hink it indicate unstable.
train_aug = A.Compose([\n A.HorizontalFlip(=0.5),\n A.VerticalFlip(=0.5),\n A.RandomBrightness(=0.1, =0.7),\n A.OneOf([\n A.GaussNoise(var_limit=[10, 50]),\n A.GaussianBlur(),\n A.MotionBlur(),\n A.MedianBlur(=3),\n ], =0.4),\n A.OneOf([\n A.GridDistortion(=5, =0.3, =1.0),\n A.OpticalDistortion(=1., =1.0)\n ],=0.2),\n A.ShiftScaleRotate(=0.7, =0.5, =0.2, =30),\n A.CoarseDropout(=1, =0.25, =0.25),\n ToTensorV2(=)\n ])\nInference is xy, xz, yz axis, and crop 512, stride 256.
\nProbability threshold(=sigmoid output). I didn't use percentile which method used many public notebook and past segmentation competiton(e.g. Volcano).
\nBecause I checked percentile threshold in local cv, it's not stable. I didn't use it.
https://www.kaggle.com/code/tereka/simpleunet-xy-xz-yz-v2-nbp-b749ff/notebook
", "rawMarkdown": "Thank you for host, I learned a lot from this competiton!\nalso special thanks to @hengck23 . I refer @hengck23 discussion topics many times.\n\n# TL;DR\n- MaxVit tiny\n- xy, xz, yz inference\n- heavy regularization\n- probability threshold\n\n# Solution\n## Model\n- MaxVit tiny\n\n```\nclass SenUNetStem(nn.Module):\n def __init__(self, encoder_name=\"resnest26d\",output_stride=32,\n encoder_depth=5 , in_chans=1,\n decoder_use_batchnorm: bool = True,\n decoder_channels: List[int] = (256, 128, 64, 32, 16),\n decoder_attention_type: Optional[str] = None, classes=1, activation=None):\n super(SenUNetStem, self).__init__()\n kwargs = dict(\n in_chans=in_chans,\n features_only=True,\n # output_stride=output_stride,\n pretrained=True,\n out_indices=tuple(range(encoder_depth)),\n )\n self.conv_stem = Conv2dReLU(in_chans, 16, 3, use_layernorm=False, padding=1)\n self.encoder = timm.create_model(encoder_name, **kwargs)\n self._out_channels = [\n 32,\n ] + self.encoder.feature_info.channels()\n\n self.decoder = UnetDecoder(\n encoder_channels=self._out_channels,\n decoder_channels=decoder_channels,\n n_blocks=encoder_depth,\n use_batchnorm=decoder_use_batchnorm,\n center=True if encoder_name.startswith(\"vgg\") else False,\n attention_type=decoder_attention_type,\n )\n\n self.segmentation_head = SegmentationHead(\n in_channels=decoder_channels[-1] + 16,\n out_channels=classes,\n activation=activation,\n kernel_size=3,\n )\n\n self.n_time = n_time\n self.pickup_index = pickup_index\n\n def forward(self, x):\n B, C, H, W = x.shape\n h = (H//32)*32\n w = (W//32)*32\n x = x[:,:,:h,:w]\n stem = self.conv_stem(x)\n features = self.encoder(x) \n features = [\n stem,\n ] + features\n\n decoder_output = self.decoder(*features)\n masks = self.segmentation_head(decoder_output)\n masks = F.pad(masks,[0,W-w,0,H-h,0,0,0,0], mode='constant', value=0)\n \n return masks[:,0]\n\nSenUNetStem(\n encoder_name=\"maxvit_tiny_tf_512.in1k\",\n classes=1,\n activation=None,\n )\n```\n\n## Dataset\n- kidney1 and 3 dataset.\n\n## Training Tricks\nI focus on regularization trick.\nbecause this competiton have unstable cv, public is not related.\nMoreover host shared public/Private LB information images, I hink it indicate unstable.\n\n- EMA\n- 50epochs\n- AdamW(Weight Decay 1e-2)\n- CutMix(until 25ep)\n- MixUp(until 25ep)\n- DiceLoss(smooth_factor=0.1)\n- Heavy Augmentation \n```\n train_aug = A.Compose([\n A.HorizontalFlip(p=0.5),\n A.VerticalFlip(p=0.5),\n A.RandomBrightness(limit=0.1, p=0.7),\n A.OneOf([\n A.GaussNoise(var_limit=[10, 50]),\n A.GaussianBlur(),\n A.MotionBlur(),\n A.MedianBlur(blur_limit=3),\n ], p=0.4),\n A.OneOf([\n A.GridDistortion(num_steps=5, distort_limit=0.3, p=1.0),\n A.OpticalDistortion(distort_limit=1., p=1.0)\n ],p=0.2),\n A.ShiftScaleRotate(p=0.7, scale_limit=0.5, shift_limit=0.2, rotate_limit=30),\n A.CoarseDropout(max_holes=1, max_height=0.25, max_width=0.25),\n ToTensorV2(transpose_mask=True)\n ])\n```\n- Crop(512)\n\n## Inference\nInference is xy, xz, yz axis, and crop 512, stride 256.\n\n## Post-Process\nProbability threshold(=sigmoid output). I didn't use percentile which method used many public notebook and past segmentation competiton(e.g. Volcano).\nBecause I checked percentile threshold in local cv, it's not stable. I didn't use it.\n\n## Not worked\n- Bigger models(maxvit base, small)\n- large size inference(1024), 512 is enough for this competiton.\n- Rotate90\n- Pretrained Other volumes(kidney_2/kidney_1_volumes)\n\nhttps://www.kaggle.com/code/tereka/simpleunet-xy-xz-yz-v2-nbp-b749ff/notebook", "votes": null }, { "id": "2640670", "postDate": "02/07/2024 03:32:21", "content": "Congratulations! Would you consider sharing your notebook?
", "rawMarkdown": "Congratulations! Would you consider sharing your notebook?", "votes": null }, { "id": "2641491", "postDate": "02/07/2024 14:11:32", "content": "@tereka congratz with gold zone and thank you for the code snippets. Could you share your ema implementation if it is not a secret, please? I've been running standard pytorch EMA implementation and was boosting the performance for CV, I wonder I did it correctly. This was my code:
\n torch_ema ExponentialMovingAverage\n\nema = ExponentialMovingAverage(model.parameters(), decay=)\n\n\n output = model(images, qmin, qmax)\n loss = criterion(output, masks)\n optimizer.zero_grad()\n loss .backward()\n optimizer.step()\n ema.update()\nthanks, I will share inference notebook after finalize.
", "rawMarkdown": "thanks, I will share inference notebook after finalize.", "votes": null }, { "id": "2641635", "postDate": "02/07/2024 15:28:07", "content": "Thanks, here is my EMA code.
\nI think your code(ema update position) is correct.
# PyTorch Lightning Module\n def training:\n self.ema_model.update(self.model)\n\n :\n def :\n super(ModelEMA, self).\n # make a copy the model accumulating moving average weights\n self. = deepcopy(model)\n self..eval\n self.decay = decay\n self.device = device # perform ema on different device from model set\n self.device is not None:\n self..(device=device)\n\n def :\n torch.no:\n ema_v, model_v zip(self..state.values, model.state.values):\n self.device is not None:\n model_v = model_v.(device=self.device)\n ema_v.copy)\n\n def update(self, model):\n self.m)\n\n def set(self, model):\n self.\n@tereka wow, I did not expect pytorch-lightning that's exactly what I needed. Thank you, you are the man!
", "rawMarkdown": "tereka wow, I did not expect pytorch-lightning that's exactly what I needed. Thank you, you are the man!", "votes": null }, { "id": "2642444", "postDate": "02/08/2024 06:40:47", "content": "@tereka , thank you for sharing the idea + code and congratulation with your 9th position.
\nI would like to ask one very beginner's question confused me a while:
\nHow do you decide change the decoder to UnetDecoder? Compared with MaxVit's original decoder, what is the benefit using this one?
", "rawMarkdown": "tereka , thank you for sharing the idea + code and congratulation with your 9th position.\n\nI would like to ask one very beginner's question confused me a while:\n\nHow do you decide change the decoder to UnetDecoder? Compared with MaxVit's original decoder, what is the benefit using this one?", "votes": null }, { "id": "2643161", "postDate": "02/08/2024 16:54:45", "content": "thanks.
\nDoes Original decoder means smp decoder ?
\nI would like to input 1/2 x 1/2 image, original max vit 1/4.
\nI need to customize smp unet.
Thanks for sharing this class @tereka!
\nA quick general EMA question.. Does a decay rate of 0.9999 typically work well for you, or do you often tune this value?
", "rawMarkdown": "Thanks for sharing this class @tereka!\n\nA quick general EMA question.. Does a decay rate of 0.9999 typically work well for you, or do you often tune this value?", "votes": null }, { "id": "2645065", "postDate": "02/09/2024 23:01:25", "content": "@brendanartley
\nthanks! EMA decay rate is 0.99 in this competition.
\nif EMA decay value is low, model convergence is very slow..
Congratulations! Would you consider sharing your notebook?
", "votes": null, "replies": [ { "id": 2641631, "author_name": "tereka", "author_url": "", "post_date": "02/07/2024 15:25:11", "content": "thanks, I will share inference notebook after finalize.
", "votes": null, "replies": [] } ] }, { "id": 2641491, "author_name": "sergiosaharovskiy", "author_url": "", "post_date": "02/07/2024 14:11:32", "content": "@tereka congratz with gold zone and thank you for the code snippets. Could you share your ema implementation if it is not a secret, please? I've been running standard pytorch EMA implementation and was boosting the performance for CV, I wonder I did it correctly. This was my code:
\n torch_ema ExponentialMovingAverage\n\nema = ExponentialMovingAverage(model.parameters(), decay=)\n\n\n output = model(images, qmin, qmax)\n loss = criterion(output, masks)\n optimizer.zero_grad()\n loss .backward()\n optimizer.step()\n ema.update()\nThanks, here is my EMA code.
\nI think your code(ema update position) is correct.
# PyTorch Lightning Module\n def training:\n self.ema_model.update(self.model)\n\n :\n def :\n super(ModelEMA, self).\n # make a copy the model accumulating moving average weights\n self. = deepcopy(model)\n self..eval\n self.decay = decay\n self.device = device # perform ema on different device from model set\n self.device is not None:\n self..(device=device)\n\n def :\n torch.no:\n ema_v, model_v zip(self..state.values, model.state.values):\n self.device is not None:\n model_v = model_v.(device=self.device)\n ema_v.copy)\n\n def update(self, model):\n self.m)\n\n def set(self, model):\n self.\n@tereka wow, I did not expect pytorch-lightning that's exactly what I needed. Thank you, you are the man!
", "votes": null, "replies": [] }, { "id": 2644734, "author_name": "brendanartley", "author_url": "", "post_date": "02/09/2024 16:32:56", "content": "Thanks for sharing this class @tereka!
\nA quick general EMA question.. Does a decay rate of 0.9999 typically work well for you, or do you often tune this value?
", "votes": null, "replies": [ { "id": 2645065, "author_name": "tereka", "author_url": "", "post_date": "02/09/2024 23:01:25", "content": "@brendanartley
\nthanks! EMA decay rate is 0.99 in this competition.
\nif EMA decay value is low, model convergence is very slow..
@tereka , thank you for sharing the idea + code and congratulation with your 9th position.
\nI would like to ask one very beginner's question confused me a while:
\nHow do you decide change the decoder to UnetDecoder? Compared with MaxVit's original decoder, what is the benefit using this one?
", "votes": null, "replies": [ { "id": 2643161, "author_name": "tereka", "author_url": "", "post_date": "02/08/2024 16:54:45", "content": "thanks.
\nDoes Original decoder means smp decoder ?
\nI would like to input 1/2 x 1/2 image, original max vit 1/4.
\nI need to customize smp unet.