{
  "id": 174269,
  "title": "ROC around 0.5 during training",
  "url": "/competitions/siim-isic-melanoma-classification/discussion/174269",
  "author_name": "",
  "post_date": "2020-08-12T23:14:15.139000500Z",
  "votes": null,
  "comment_count": 9,
  "views": 0,
  "content": "<p>Hi ! </p>\n<p>I have a problem when I am training my data : the ROC is always around 0.5 and the network is converging very fastly. <br>\nI have tried many things like : </p>\n<ul>\n<li>Augmented data ;</li>\n<li>Stratified splited data ;</li>\n<li>Replacing Resnet by EfficientNet</li>\n<li>Using weight loss as the loss function; </li>\n<li>AdamW optimizer</li>\n<li>GPU training, lr = 0.001</li>\n<li>many things I don't remember yet…</li>\n</ul>\n<p>Here is my notebook : <a href=\"https://www.kaggle.com/anthokalel/pytorch-starting-with-efficientnet-model\" target=\"_blank\">https://www.kaggle.com/anthokalel/pytorch-starting-with-efficientnet-model</a></p>\n<p>Is one of you have hints to give me to help me rising the ROC score ?<br>\nThank you. </p>",
  "messages": [
    {
      "id": "968348",
      "postDate": "08/12/2020 23:14:15",
      "content": "<p>Hi ! </p>\n<p>I have a problem when I am training my data : the ROC is always around 0.5 and the network is converging very fastly. <br>\nI have tried many things like : </p>\n<ul>\n<li>Augmented data ;</li>\n<li>Stratified splited data ;</li>\n<li>Replacing Resnet by EfficientNet</li>\n<li>Using weight loss as the loss function; </li>\n<li>AdamW optimizer</li>\n<li>GPU training, lr = 0.001</li>\n<li>many things I don't remember yet…</li>\n</ul>\n<p>Here is my notebook : <a href=\"https://www.kaggle.com/anthokalel/pytorch-starting-with-efficientnet-model\" target=\"_blank\">https://www.kaggle.com/anthokalel/pytorch-starting-with-efficientnet-model</a></p>\n<p>Is one of you have hints to give me to help me rising the ROC score ?<br>\nThank you. </p>",
      "rawMarkdown": "Hi ! \n\nI have a problem when I am training my data : the ROC is always around 0.5 and the network is converging very fastly. \nI have tried many things like : \n\n- Augmented data ;\n- Stratified splited data ;\n- Replacing Resnet by EfficientNet\n- Using weight loss as the loss function; \n- AdamW optimizer\n- GPU training, lr = 0.001\n- many things I don't remember yet...\n\nHere is my notebook : https://www.kaggle.com/anthokalel/pytorch-starting-with-efficientnet-model\n\nIs one of you have hints to give me to help me rising the ROC score ?\nThank you.",
      "votes": null
    },
    {
      "id": "968387",
      "postDate": "08/13/2020 00:53:35",
      "content": "<p>I couldn't open your link. LR seems high. Try 0.0001 or lower to start off.</p>",
      "rawMarkdown": "I couldn't open your link. LR seems high. Try 0.0001 or lower to start off.",
      "votes": null
    },
    {
      "id": "968403",
      "postDate": "08/13/2020 01:29:38",
      "content": "<p>Reducing the learning rate may help as <a href=\"https://www.kaggle.com/richardepstein\" target=\"_blank\">@richardepstein</a> already mentioned. One thing that helped me was undersampling the data. I ended up using 9 thousand samples for each class (using some from previous years). </p>\n<p>Hope this helps!</p>",
      "rawMarkdown": "Reducing the learning rate may help as @richardepstein already mentioned. One thing that helped me was undersampling the data. I ended up using 9 thousand samples for each class (using some from previous years). \n\nHope this helps!",
      "votes": null
    },
    {
      "id": "968830",
      "postDate": "08/13/2020 09:10:30",
      "content": "<p>I had similar experiences : whether you keep the LR and change loss to regular binary cross entropy maybe add some label smoothing or you lower the learning rate to 1e-5 and keep focal loss. </p>",
      "rawMarkdown": "I had similar experiences : whether you keep the LR and change loss to regular binary cross entropy maybe add some label smoothing or you lower the learning rate to 1e-5 and keep focal loss.",
      "votes": null
    },
    {
      "id": "969096",
      "postDate": "08/13/2020 13:03:40",
      "content": "<p>Thank you for your answer. I tried to change the loss function and using binary cross entropy or focal loss but it didn't work… </p>",
      "rawMarkdown": "Thank you for your answer. I tried to change the loss function and using binary cross entropy or focal loss but it didn't work...",
      "votes": null
    },
    {
      "id": "969099",
      "postDate": "08/13/2020 13:04:48",
      "content": "<p>Thank you for your answer. I think the link is working now. I tried to low the LR but the ROC is not evolving…</p>",
      "rawMarkdown": "Thank you for your answer. I think the link is working now. I tried to low the LR but the ROC is not evolving...",
      "votes": null
    },
    {
      "id": "969100",
      "postDate": "08/13/2020 13:05:42",
      "content": "<p>I'm going to try undersampling the data. Thank you ! </p>",
      "rawMarkdown": "I'm going to try undersampling the data. Thank you !",
      "votes": null
    },
    {
      "id": "969152",
      "postDate": "08/13/2020 13:50:19",
      "content": "<p>This usually happens when your target and prediction tensors are of different dimensions. You should carefully check that.</p>",
      "rawMarkdown": "This usually happens when your target and prediction tensors are of different dimensions. You should carefully check that.",
      "votes": null
    },
    {
      "id": "969162",
      "postDate": "08/13/2020 14:00:48",
      "content": "<p>They have the same dimensions : [32, 1]…</p>",
      "rawMarkdown": "They have the same dimensions : [32, 1]...",
      "votes": null
    },
    {
      "id": "971070",
      "postDate": "08/15/2020 06:41:34",
      "content": "<p>Default weight decay of AdamW in pytorch is very high, try decreasing that to less than 1e-3 or 1e-4, I had similar problem with keras. Weight decay of 1e-5 worked best for me</p>",
      "rawMarkdown": "Default weight decay of AdamW in pytorch is very high, try decreasing that to less than 1e-3 or 1e-4, I had similar problem with keras. Weight decay of 1e-5 worked best for me",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 968387,
      "author_name": "richardepstein",
      "author_url": "",
      "post_date": "08/13/2020 00:53:35",
      "content": "<p>I couldn't open your link. LR seems high. Try 0.0001 or lower to start off.</p>",
      "votes": null,
      "replies": [
        {
          "id": 969099,
          "author_name": "anthokalel",
          "author_url": "",
          "post_date": "08/13/2020 13:04:48",
          "content": "<p>Thank you for your answer. I think the link is working now. I tried to low the LR but the ROC is not evolving…</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 968403,
      "author_name": "saumandas",
      "author_url": "",
      "post_date": "08/13/2020 01:29:38",
      "content": "<p>Reducing the learning rate may help as <a href=\"https://www.kaggle.com/richardepstein\" target=\"_blank\">@richardepstein</a> already mentioned. One thing that helped me was undersampling the data. I ended up using 9 thousand samples for each class (using some from previous years). </p>\n<p>Hope this helps!</p>",
      "votes": null,
      "replies": [
        {
          "id": 969100,
          "author_name": "anthokalel",
          "author_url": "",
          "post_date": "08/13/2020 13:05:42",
          "content": "<p>I'm going to try undersampling the data. Thank you ! </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 968830,
      "author_name": "aziz69",
      "author_url": "",
      "post_date": "08/13/2020 09:10:30",
      "content": "<p>I had similar experiences : whether you keep the LR and change loss to regular binary cross entropy maybe add some label smoothing or you lower the learning rate to 1e-5 and keep focal loss. </p>",
      "votes": null,
      "replies": [
        {
          "id": 969096,
          "author_name": "anthokalel",
          "author_url": "",
          "post_date": "08/13/2020 13:03:40",
          "content": "<p>Thank you for your answer. I tried to change the loss function and using binary cross entropy or focal loss but it didn't work… </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 969152,
      "author_name": "philippsinger",
      "author_url": "",
      "post_date": "08/13/2020 13:50:19",
      "content": "<p>This usually happens when your target and prediction tensors are of different dimensions. You should carefully check that.</p>",
      "votes": null,
      "replies": [
        {
          "id": 969162,
          "author_name": "anthokalel",
          "author_url": "",
          "post_date": "08/13/2020 14:00:48",
          "content": "<p>They have the same dimensions : [32, 1]…</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 971070,
      "author_name": "vignet",
      "author_url": "",
      "post_date": "08/15/2020 06:41:34",
      "content": "<p>Default weight decay of AdamW in pytorch is very high, try decreasing that to less than 1e-3 or 1e-4, I had similar problem with keras. Weight decay of 1e-5 worked best for me</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "968348": "Hi ! \n\nI have a problem when I am training my data : the ROC is always around 0.5 and the network is converging very fastly. \nI have tried many things like : \n\n- Augmented data ;\n- Stratified splited data ;\n- Replacing Resnet by EfficientNet\n- Using weight loss as the loss function; \n- AdamW optimizer\n- GPU training, lr = 0.001\n- many things I don't remember yet...\n\nHere is my notebook : https://www.kaggle.com/anthokalel/pytorch-starting-with-efficientnet-model\n\nIs one of you have hints to give me to help me rising the ROC score ?\nThank you.",
    "968387": "I couldn't open your link. LR seems high. Try 0.0001 or lower to start off.",
    "968403": "Reducing the learning rate may help as @richardepstein already mentioned. One thing that helped me was undersampling the data. I ended up using 9 thousand samples for each class (using some from previous years). \n\nHope this helps!",
    "968830": "I had similar experiences : whether you keep the LR and change loss to regular binary cross entropy maybe add some label smoothing or you lower the learning rate to 1e-5 and keep focal loss.",
    "969096": "Thank you for your answer. I tried to change the loss function and using binary cross entropy or focal loss but it didn't work...",
    "969099": "Thank you for your answer. I think the link is working now. I tried to low the LR but the ROC is not evolving...",
    "969100": "I'm going to try undersampling the data. Thank you !",
    "969152": "This usually happens when your target and prediction tensors are of different dimensions. You should carefully check that.",
    "969162": "They have the same dimensions : [32, 1]...",
    "971070": "Default weight decay of AdamW in pytorch is very high, try decreasing that to less than 1e-3 or 1e-4, I had similar problem with keras. Weight decay of 1e-5 worked best for me"
  },
  "source": "meta"
}