Hi guys, I am trying to use ViT model and 5-fold cross validation in this competition, but it seems easy to get overfitting, the train loss(blue) and val loss(red) work will at train dataset, but only got 0.72 at leaderborad, is there any solution?
\nTHANKS!
\n:)
Hi guys, I am trying to use ViT model and 5-fold cross validation in this competition, but it seems easy to get overfitting, the train loss(blue) and val loss(red) work will at train dataset, but only got 0.72 at leaderborad, is there any solution?
\nTHANKS!
\n:)
There is probably a bug in your implementation, is difficult to say without a look in the source code.
", "rawMarkdown": "There is probably a bug in your implementation, is difficult to say without a look in the source code.", "votes": null }, { "id": "2662687", "postDate": "02/22/2024 04:37:32", "content": "Hi, my work just replace the model from effectivenet (https://www.kaggle.com/code/alejopaullier/hms-efficientnetb0-pytorch-train) to ViT and used reshape the images.
\nHere are codes i changed:
\n (nn.Module):\n ():\n (CustomModel, self).__init__()\n self.USE_KAGGLE_SPECTROGRAMS = \n self.USE_EEG_SPECTROGRAMS = \n self.model = timm.create_model(\n , \n pretrained=,\n drop_rate=,\n drop_path_rate=,\n attn_drop_rate = ,\n )\n\n self.checkpoint = torch.load()\n self.model.load_state_dict(self.checkpoint[])\n k,v self.model.named_parameters():\n\n v.requires_grad = \n num_features = self.model.head.in_features\n self.model.head = nn.Linear(num_features, num_classes)\n ():\n \n \n spectrograms = [x[:, :, :, i:i+] i ()]\n spectrograms = torch.cat(spectrograms, dim=)\n \n eegs = [x[:, :, :, i:i+] i (, )]\n eegs = torch.cat(eegs, dim=)\n \n self.USE_KAGGLE_SPECTROGRAMS & self.USE_EEG_SPECTROGRAMS:\n x = torch.cat([spectrograms, eegs], dim=)\n self.USE_EEG_SPECTROGRAMS:\n x = eegs\n :\n x = spectrograms\n\n x = torch.cat([x, x, x], dim=)\n\n x = x.permute(, , , )\n x = F.interpolate(x, size=(, ), mode=, align_corners=)\n (x.shape)\n x\nI just can't upload the loss image…..
\nThanks a lot!!!
", "rawMarkdown": "Hi, my work just replace the model from effectivenet (https://www.kaggle.com/code/alejopaullier/hms-efficientnetb0-pytorch-train) to ViT and used reshape the images.\n\nHere are codes i changed:\n\n```python\nclass CustomModel(nn.Module):\n def __init__(self, config, num_classes: int = 6, pretrained: bool = True):\n super(CustomModel, self).__init__()\n self.USE_KAGGLE_SPECTROGRAMS = True\n self.USE_EEG_SPECTROGRAMS = True\n self.model = timm.create_model(\n \"vit_small_patch32_384\", # 使用ViT-B/16模型\n pretrained=False,\n drop_rate=0.5,\n drop_path_rate=0.5,\n attn_drop_rate = 0.5,\n )\n\n self.checkpoint = torch.load(r'/dengby/vit&swin_vit_weights/vit_small_patch32_384.pth')\n self.model.load_state_dict(self.checkpoint['model'])\n for k,v in self.model.named_parameters():\n\n v.requires_grad = True\n num_features = self.model.head.in_features\n self.model.head = nn.Linear(num_features, num_classes)\n```\n\n```python\ndef __reshape_input(self, x):\n \"\"\"\n Reshapes input (128, 256, 8) -> (512, 512, 3) monotone image.\n \"\"\" \n # === Get spectrograms ===\n spectrograms = [x[:, :, :, i:i+1] for i in range(4)]\n spectrograms = torch.cat(spectrograms, dim=1)\n # === Get EEG spectrograms ===\n eegs = [x[:, :, :, i:i+1] for i in range(4, 8)]\n eegs = torch.cat(eegs, dim=1)\n # === Reshape (512, 512, 3) ===\n if self.USE_KAGGLE_SPECTROGRAMS & self.USE_EEG_SPECTROGRAMS:\n x = torch.cat([spectrograms, eegs], dim=2)\n elif self.USE_EEG_SPECTROGRAMS:\n x = eegs\n else:\n x = spectrograms\n\n x = torch.cat([x, x, x], dim=3)\n\n x = x.permute(0, 3, 2, 1)\n x = F.interpolate(x, size=(384, 384), mode='bilinear', align_corners=False)\n print (x.shape)\n return x\n```\nI just can't upload the loss image.....\n\n\nThanks a lot!!!", "votes": null }, { "id": "2663042", "postDate": "02/22/2024 09:34:28", "content": "Do not use vit_small_patch32_384, this requires additional processing 512→384. You can use tiny_vit_21m_512 or maxvit_tiny_tf_512, they have fewer parameters and receive an image size of 512.
\nMy best scores:
\nefficientnet_b0 LB 0.41
\ntiny_vit_21m_512 LB 0.4
thanks!!!
\nWhich means reshape will highly affect results? Or large input image size can contain more information?
I think they are all possible.
", "rawMarkdown": "I think they are all possible.", "votes": null }, { "id": "2663133", "postDate": "02/22/2024 10:26:50", "content": "Thanks, I will try it later!
", "rawMarkdown": "Thanks, I will try it later!", "votes": null }, { "id": "2663339", "postDate": "02/22/2024 12:38:13", "content": "Sorry for the delay in answering, but i think you got what you needed 😃 Good luck @rexdeng
", "rawMarkdown": "Sorry for the delay in answering, but i think you got what you needed 😃 Good luck @rexdeng", "votes": null }, { "id": "2666815", "postDate": "02/24/2024 16:49:55", "content": "Try to use larger size input and watch out for the model architecture before using any large input size
", "rawMarkdown": "Try to use larger size input and watch out for the model architecture before using any large input size", "votes": null }, { "id": "2669186", "postDate": "02/26/2024 06:12:41", "content": "Thanks! I will try it!
\n:)
Seems like the complexity of the model seems to have a significant impact on the accuracy. Complex models are prone to overfitting, which can lead to good performance on the training and validation data, but poor performance on the leaderboard.
", "rawMarkdown": "Seems like the complexity of the model seems to have a significant impact on the accuracy. Complex models are prone to overfitting, which can lead to good performance on the training and validation data, but poor performance on the leaderboard.", "votes": null } ], "comments": [ { "id": 2662618, "author_name": "gabrielfreddi", "author_url": "", "post_date": "02/22/2024 03:01:57", "content": "There is probably a bug in your implementation, is difficult to say without a look in the source code.
", "votes": null, "replies": [ { "id": 2662687, "author_name": "rexdeng", "author_url": "", "post_date": "02/22/2024 04:37:32", "content": "Hi, my work just replace the model from effectivenet (https://www.kaggle.com/code/alejopaullier/hms-efficientnetb0-pytorch-train) to ViT and used reshape the images.
\nHere are codes i changed:
\n (nn.Module):\n ():\n (CustomModel, self).__init__()\n self.USE_KAGGLE_SPECTROGRAMS = \n self.USE_EEG_SPECTROGRAMS = \n self.model = timm.create_model(\n , \n pretrained=,\n drop_rate=,\n drop_path_rate=,\n attn_drop_rate = ,\n )\n\n self.checkpoint = torch.load()\n self.model.load_state_dict(self.checkpoint[])\n k,v self.model.named_parameters():\n\n v.requires_grad = \n num_features = self.model.head.in_features\n self.model.head = nn.Linear(num_features, num_classes)\n ():\n \n \n spectrograms = [x[:, :, :, i:i+] i ()]\n spectrograms = torch.cat(spectrograms, dim=)\n \n eegs = [x[:, :, :, i:i+] i (, )]\n eegs = torch.cat(eegs, dim=)\n \n self.USE_KAGGLE_SPECTROGRAMS & self.USE_EEG_SPECTROGRAMS:\n x = torch.cat([spectrograms, eegs], dim=)\n self.USE_EEG_SPECTROGRAMS:\n x = eegs\n :\n x = spectrograms\n\n x = torch.cat([x, x, x], dim=)\n\n x = x.permute(, , , )\n x = F.interpolate(x, size=(, ), mode=, align_corners=)\n (x.shape)\n x\nI just can't upload the loss image…..
\nThanks a lot!!!
", "votes": null, "replies": [ { "id": 2663339, "author_name": "gabrielfreddi", "author_url": "", "post_date": "02/22/2024 12:38:13", "content": "Sorry for the delay in answering, but i think you got what you needed 😃 Good luck @rexdeng
", "votes": null, "replies": [] } ] } ] }, { "id": 2663042, "author_name": "zznznb", "author_url": "", "post_date": "02/22/2024 09:34:28", "content": "Do not use vit_small_patch32_384, this requires additional processing 512→384. You can use tiny_vit_21m_512 or maxvit_tiny_tf_512, they have fewer parameters and receive an image size of 512.
\nMy best scores:
\nefficientnet_b0 LB 0.41
\ntiny_vit_21m_512 LB 0.4
thanks!!!
\nWhich means reshape will highly affect results? Or large input image size can contain more information?
I think they are all possible.
", "votes": null, "replies": [ { "id": 2663133, "author_name": "rexdeng", "author_url": "", "post_date": "02/22/2024 10:26:50", "content": "Thanks, I will try it later!
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2666815, "author_name": "ahmedtambal", "author_url": "", "post_date": "02/24/2024 16:49:55", "content": "Try to use larger size input and watch out for the model architecture before using any large input size
", "votes": null, "replies": [ { "id": 2669186, "author_name": "rexdeng", "author_url": "", "post_date": "02/26/2024 06:12:41", "content": "Thanks! I will try it!
\n:)
Seems like the complexity of the model seems to have a significant impact on the accuracy. Complex models are prone to overfitting, which can lead to good performance on the training and validation data, but poor performance on the leaderboard.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2662565": "Hi guys, I am trying to use ViT model and 5-fold cross validation in this competition, but it seems easy to get overfitting, the train loss(blue) and val loss(red) work will at train dataset, but only got 0.72 at leaderborad, is there any solution?\nTHANKS!\n:)", "2662618": "There is probably a bug in your implementation, is difficult to say without a look in the source code.", "2662687": "Hi, my work just replace the model from effectivenet (https://www.kaggle.com/code/alejopaullier/hms-efficientnetb0-pytorch-train) to ViT and used reshape the images.\n\nHere are codes i changed:\n\n```python\nclass CustomModel(nn.Module):\n def __init__(self, config, num_classes: int = 6, pretrained: bool = True):\n super(CustomModel, self).__init__()\n self.USE_KAGGLE_SPECTROGRAMS = True\n self.USE_EEG_SPECTROGRAMS = True\n self.model = timm.create_model(\n \"vit_small_patch32_384\", # 使用ViT-B/16模型\n pretrained=False,\n drop_rate=0.5,\n drop_path_rate=0.5,\n attn_drop_rate = 0.5,\n )\n\n self.checkpoint = torch.load(r'/dengby/vit&swin_vit_weights/vit_small_patch32_384.pth')\n self.model.load_state_dict(self.checkpoint['model'])\n for k,v in self.model.named_parameters():\n\n v.requires_grad = True\n num_features = self.model.head.in_features\n self.model.head = nn.Linear(num_features, num_classes)\n```\n\n```python\ndef __reshape_input(self, x):\n \"\"\"\n Reshapes input (128, 256, 8) -> (512, 512, 3) monotone image.\n \"\"\" \n # === Get spectrograms ===\n spectrograms = [x[:, :, :, i:i+1] for i in range(4)]\n spectrograms = torch.cat(spectrograms, dim=1)\n # === Get EEG spectrograms ===\n eegs = [x[:, :, :, i:i+1] for i in range(4, 8)]\n eegs = torch.cat(eegs, dim=1)\n # === Reshape (512, 512, 3) ===\n if self.USE_KAGGLE_SPECTROGRAMS & self.USE_EEG_SPECTROGRAMS:\n x = torch.cat([spectrograms, eegs], dim=2)\n elif self.USE_EEG_SPECTROGRAMS:\n x = eegs\n else:\n x = spectrograms\n\n x = torch.cat([x, x, x], dim=3)\n\n x = x.permute(0, 3, 2, 1)\n x = F.interpolate(x, size=(384, 384), mode='bilinear', align_corners=False)\n print (x.shape)\n return x\n```\nI just can't upload the loss image.....\n\n\nThanks a lot!!!", "2663042": "Do not use **vit_small_patch32_384**, this requires additional processing 512→384. You can use **tiny_vit_21m_512** or **maxvit_tiny_tf_512**, they have fewer parameters and receive an image size of 512.\nMy best scores: \nefficientnet_b0 LB 0.41\ntiny_vit_21m_512 LB 0.4", "2663088": "thanks!!!\nWhich means reshape will highly affect results? Or large input image size can contain more information?", "2663096": "I think they are all possible.", "2663133": "Thanks, I will try it later!", "2663339": "Sorry for the delay in answering, but i think you got what you needed 😃 Good luck @rexdeng", "2666815": "Try to use larger size input and watch out for the model architecture before using any large input size", "2669186": "Thanks! I will try it!\n:)", "2669333": "Seems like the complexity of the model seems to have a significant impact on the accuracy. Complex models are prone to overfitting, which can lead to good performance on the training and validation data, but poor performance on the leaderboard." }, "source": "meta" }