{
  "id": 534562,
  "title": "[need help]why my notebook loss dont reduce",
  "url": "/competitions/rsna-2024-lumbar-spine-degenerative-classification/discussion/534562",
  "author_name": "",
  "post_date": "2024-09-17T11:21:29.480263300Z",
  "votes": null,
  "comment_count": 4,
  "views": 0,
  "content": "<p>i have tried many way to reduce my loss,however it doesnt work. i just a beginner of Deeplearning,so it is a great pleasure if you can help me to find out my false.maybe my notebook is very hard to read, but i really need you help<br>\nnotebook:<a href=\"https://www.kaggle.com/code/ottolumous/rsna-with-vit\" target=\"_blank\">https://www.kaggle.com/code/ottolumous/rsna-with-vit</a></p>",
  "messages": [
    {
      "id": "2991350",
      "postDate": "09/17/2024 11:21:29",
      "content": "<p>i have tried many way to reduce my loss,however it doesnt work. i just a beginner of Deeplearning,so it is a great pleasure if you can help me to find out my false.maybe my notebook is very hard to read, but i really need you help<br>\nnotebook:<a href=\"https://www.kaggle.com/code/ottolumous/rsna-with-vit\" target=\"_blank\">https://www.kaggle.com/code/ottolumous/rsna-with-vit</a></p>",
      "rawMarkdown": "i have tried many way to reduce my loss,however it doesnt work. i just a beginner of Deeplearning,so it is a great pleasure if you can help me to find out my false.maybe my notebook is very hard to read, but i really need you help\nnotebook:https://www.kaggle.com/code/ottolumous/rsna-with-vit",
      "votes": null
    },
    {
      "id": "2991354",
      "postDate": "09/17/2024 11:26:05",
      "content": "<p>i just want to try a new way to solve the problem in order to improve skill of coding.so i use vit</p>",
      "rawMarkdown": "i just want to try a new way to solve the problem in order to improve skill of coding.so i use vit",
      "votes": null
    },
    {
      "id": "2991400",
      "postDate": "09/17/2024 12:12:11",
      "content": "<p>the larger problem is that when i training,my loss will reduce very fast in first however it may become larger the next moment like 1.22,0.99,0.45,0.22,0.13,1.34 just like this </p>",
      "rawMarkdown": "the larger problem is that when i training,my loss will reduce very fast in first however it may become larger the next moment like 1.22,0.99,0.45,0.22,0.13,1.34 just like this",
      "votes": null
    },
    {
      "id": "2992894",
      "postDate": "09/19/2024 08:20:35",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/ottolumous\" target=\"_blank\">@ottolumous</a>,</p>\n<p>I cannot see your notebook, but here are few pointers to consider.<br>\nIt might be issue with your learning rate.<br>\nIf you choose learning rate too high then model loss will decrease very fast and it may bounce around minima (probably local minima).<br>\nYou can experiment with different learning rates (good starting points are 0.01,0.001, 0.0001 if you are training ViT from scratch.</p>\n<p>your loss behaviour indicates you are using high learning rate, try reducing it.</p>\n<p>Apart from learning rate search for learning rate schedulers used in model training.<br>\ncosine schedulers are great way to adjust learning rate in model training , you can get libraries for the same and also public notebooks utilizing cosine learning rate schedule.</p>\n<p>ViTs are hard to train and can take longer to generalize well.<br>\nyou will need to regularize the model using l2 or l1 regularization.</p>\n<p>L2 Regularization can be controlled with weight decay parameter in AdamW optimizer (similarly you can find the same for other optimizers as well.)</p>\n<p>Try these things and report back any progress and further  issues you face.</p>",
      "rawMarkdown": "Hi @ottolumous,\n\nI cannot see your notebook, but here are few pointers to consider.\nIt might be issue with your learning rate.\nIf you choose learning rate too high then model loss will decrease very fast and it may bounce around minima (probably local minima).\nYou can experiment with different learning rates (good starting points are 0.01,0.001, 0.0001 if you are training ViT from scratch.\n\nyour loss behaviour indicates you are using high learning rate, try reducing it.\n\nApart from learning rate search for learning rate schedulers used in model training.\ncosine schedulers are great way to adjust learning rate in model training , you can get libraries for the same and also public notebooks utilizing cosine learning rate schedule.\n\nViTs are hard to train and can take longer to generalize well.\nyou will need to regularize the model using l2 or l1 regularization.\n\nL2 Regularization can be controlled with weight decay parameter in AdamW optimizer (similarly you can find the same for other optimizers as well.)\n\nTry these things and report back any progress and further  issues you face.",
      "votes": null
    },
    {
      "id": "2993005",
      "postDate": "09/19/2024 11:12:47",
      "content": "<p>ok.really really thanks you.i have tried to reduce my lr before.but i did not work,it still go up and down so i think there maybe some problem with my datacode.if you can help me check for another time, i will very appreciate it. <br>\n   <a href=\"https://www.kaggle.com/code/ottolumous/rsna-with-vit\" target=\"_blank\">https://www.kaggle.com/code/ottolumous/rsna-with-vit</a><br>\nreally really thanks you!!!!</p>",
      "rawMarkdown": "ok.really really thanks you.i have tried to reduce my lr before.but i did not work,it still go up and down so i think there maybe some problem with my datacode.if you can help me check for another time, i will very appreciate it. \n   https://www.kaggle.com/code/ottolumous/rsna-with-vit\nreally really thanks you!!!!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2991354,
      "author_name": "ottolumous",
      "author_url": "",
      "post_date": "09/17/2024 11:26:05",
      "content": "<p>i just want to try a new way to solve the problem in order to improve skill of coding.so i use vit</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2991400,
      "author_name": "ottolumous",
      "author_url": "",
      "post_date": "09/17/2024 12:12:11",
      "content": "<p>the larger problem is that when i training,my loss will reduce very fast in first however it may become larger the next moment like 1.22,0.99,0.45,0.22,0.13,1.34 just like this </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2992894,
      "author_name": "rohitchaudhari25",
      "author_url": "",
      "post_date": "09/19/2024 08:20:35",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/ottolumous\" target=\"_blank\">@ottolumous</a>,</p>\n<p>I cannot see your notebook, but here are few pointers to consider.<br>\nIt might be issue with your learning rate.<br>\nIf you choose learning rate too high then model loss will decrease very fast and it may bounce around minima (probably local minima).<br>\nYou can experiment with different learning rates (good starting points are 0.01,0.001, 0.0001 if you are training ViT from scratch.</p>\n<p>your loss behaviour indicates you are using high learning rate, try reducing it.</p>\n<p>Apart from learning rate search for learning rate schedulers used in model training.<br>\ncosine schedulers are great way to adjust learning rate in model training , you can get libraries for the same and also public notebooks utilizing cosine learning rate schedule.</p>\n<p>ViTs are hard to train and can take longer to generalize well.<br>\nyou will need to regularize the model using l2 or l1 regularization.</p>\n<p>L2 Regularization can be controlled with weight decay parameter in AdamW optimizer (similarly you can find the same for other optimizers as well.)</p>\n<p>Try these things and report back any progress and further  issues you face.</p>",
      "votes": null,
      "replies": [
        {
          "id": 2993005,
          "author_name": "ottolumous",
          "author_url": "",
          "post_date": "09/19/2024 11:12:47",
          "content": "<p>ok.really really thanks you.i have tried to reduce my lr before.but i did not work,it still go up and down so i think there maybe some problem with my datacode.if you can help me check for another time, i will very appreciate it. <br>\n   <a href=\"https://www.kaggle.com/code/ottolumous/rsna-with-vit\" target=\"_blank\">https://www.kaggle.com/code/ottolumous/rsna-with-vit</a><br>\nreally really thanks you!!!!</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2991350": "i have tried many way to reduce my loss,however it doesnt work. i just a beginner of Deeplearning,so it is a great pleasure if you can help me to find out my false.maybe my notebook is very hard to read, but i really need you help\nnotebook:https://www.kaggle.com/code/ottolumous/rsna-with-vit",
    "2991354": "i just want to try a new way to solve the problem in order to improve skill of coding.so i use vit",
    "2991400": "the larger problem is that when i training,my loss will reduce very fast in first however it may become larger the next moment like 1.22,0.99,0.45,0.22,0.13,1.34 just like this",
    "2992894": "Hi @ottolumous,\n\nI cannot see your notebook, but here are few pointers to consider.\nIt might be issue with your learning rate.\nIf you choose learning rate too high then model loss will decrease very fast and it may bounce around minima (probably local minima).\nYou can experiment with different learning rates (good starting points are 0.01,0.001, 0.0001 if you are training ViT from scratch.\n\nyour loss behaviour indicates you are using high learning rate, try reducing it.\n\nApart from learning rate search for learning rate schedulers used in model training.\ncosine schedulers are great way to adjust learning rate in model training , you can get libraries for the same and also public notebooks utilizing cosine learning rate schedule.\n\nViTs are hard to train and can take longer to generalize well.\nyou will need to regularize the model using l2 or l1 regularization.\n\nL2 Regularization can be controlled with weight decay parameter in AdamW optimizer (similarly you can find the same for other optimizers as well.)\n\nTry these things and report back any progress and further  issues you face.",
    "2993005": "ok.really really thanks you.i have tried to reduce my lr before.but i did not work,it still go up and down so i think there maybe some problem with my datacode.if you can help me check for another time, i will very appreciate it. \n   https://www.kaggle.com/code/ottolumous/rsna-with-vit\nreally really thanks you!!!!"
  },
  "source": "meta"
}