{
  "id": 192550,
  "title": "confidences should sum to 1 is killing me ...",
  "url": "/competitions/lyft-motion-prediction-autonomous-vehicles/discussion/192550",
  "author_name": "",
  "post_date": "2020-10-22T06:40:08.880495400Z",
  "votes": 4,
  "comment_count": 9,
  "views": 0,
  "content": "<p>I want to develop my new models, not fixing this bug..</p>",
  "messages": [
    {
      "id": "1056859",
      "postDate": "10/22/2020 06:40:08",
      "content": "<p>I want to develop my new models, not fixing this bug..</p>",
      "rawMarkdown": "I want to develop my new models, not fixing this bug..",
      "votes": null
    },
    {
      "id": "1056969",
      "postDate": "10/22/2020 08:23:09",
      "content": "<p>Clipping grad norm and adjusting learning rate is working now. <br>\nPlease don't stop until tomorrow..</p>",
      "rawMarkdown": "Clipping grad norm and adjusting learning rate is working now. \nPlease don't stop until tomorrow..",
      "votes": null
    },
    {
      "id": "1057107",
      "postDate": "10/22/2020 11:21:13",
      "content": "<p><a href=\"https://www.kaggle.com/youhanlee\" target=\"_blank\">@youhanlee</a> you can refer to part 3 of following post</p>\n<p><a href=\"https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/191599\" target=\"_blank\">https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/191599</a> </p>",
      "rawMarkdown": "youhanlee you can refer to part 3 of following post\n\nhttps://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/191599",
      "votes": null
    },
    {
      "id": "1057147",
      "postDate": "10/22/2020 12:17:10",
      "content": "<p>What did you clip norms at? I never had this issue, but did occasionally get a massive spike in loss/grad, so I just clipped at 1. </p>",
      "rawMarkdown": "What did you clip norms at? I never had this issue, but did occasionally get a massive spike in loss/grad, so I just clipped at 1.",
      "votes": null
    },
    {
      "id": "1057168",
      "postDate": "10/22/2020 12:43:41",
      "content": "<p>I clipped at 100. If this epoch is going well, I will adjust this.<br>\nUntill now, the learning is going well !!</p>",
      "rawMarkdown": "I clipped at 100. If this epoch is going well, I will adjust this.\nUntill now, the learning is going well !!",
      "votes": null
    },
    {
      "id": "1057188",
      "postDate": "10/22/2020 12:59:07",
      "content": "<p>I tried <br>\n(1) error = torch.log(confidences + 1e-10) - 0.5 * torch.sum(error, dim=-1)<br>\n(2) assert torch.allclose(torch.sum(confidences, dim=1), confidences.new_ones((batch_size,)), atol=1e-04), \"confidences should sum to 1\" - change atol to 1e04<br>\n(3) gradient clipping<br>\n(4) lower learning rate (1-e5)</p>\n<p>Only (3) works for me until now. But there would be other options to fix this issue.</p>",
      "rawMarkdown": "I tried \n(1) error = torch.log(confidences + 1e-10) - 0.5 * torch.sum(error, dim=-1)\n(2) assert torch.allclose(torch.sum(confidences, dim=1), confidences.new_ones((batch_size,)), atol=1e-04), \"confidences should sum to 1\" - change atol to 1e04\n(3) gradient clipping\n(4) lower learning rate (1-e5)\n\nOnly (3) works for me until now. But there would be other options to fix this issue.",
      "votes": null
    },
    {
      "id": "1058564",
      "postDate": "10/23/2020 21:51:01",
      "content": "<p>I posted some ideas on a different topic:<br>\n<a href=\"https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/187773\" target=\"_blank\">https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/187773</a></p>",
      "rawMarkdown": "I posted some ideas on a different topic:\nhttps://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/187773",
      "votes": null
    },
    {
      "id": "1078446",
      "postDate": "11/14/2020 19:53:36",
      "content": "<p><a href=\"https://www.kaggle.com/youhanlee\" target=\"_blank\">@youhanlee</a> still having issues with this? I am clipping the gradients but still experiencing this..</p>",
      "rawMarkdown": "youhanlee still having issues with this? I am clipping the gradients but still experiencing this..",
      "votes": null
    },
    {
      "id": "1078455",
      "postDate": "11/14/2020 20:08:55",
      "content": "<p>Did you check:<br>\n<a href=\"https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/187773\" target=\"_blank\">https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/187773</a></p>\n<p>I fixed the problem by implementing the easy \"fixes\"</p>",
      "rawMarkdown": "Did you check:\nhttps://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/187773\n\nI fixed the problem by implementing the easy \"fixes\"",
      "votes": null
    },
    {
      "id": "1084558",
      "postDate": "11/20/2020 06:37:15",
      "content": "<p>Try using logits with torch.log_softmax instead of probabilities and torch.log.</p>",
      "rawMarkdown": "Try using logits with torch.log_softmax instead of probabilities and torch.log.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1056969,
      "author_name": "youhanlee",
      "author_url": "",
      "post_date": "10/22/2020 08:23:09",
      "content": "<p>Clipping grad norm and adjusting learning rate is working now. <br>\nPlease don't stop until tomorrow..</p>",
      "votes": null,
      "replies": [
        {
          "id": 1057147,
          "author_name": "fnands",
          "author_url": "",
          "post_date": "10/22/2020 12:17:10",
          "content": "<p>What did you clip norms at? I never had this issue, but did occasionally get a massive spike in loss/grad, so I just clipped at 1. </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1057168,
          "author_name": "youhanlee",
          "author_url": "",
          "post_date": "10/22/2020 12:43:41",
          "content": "<p>I clipped at 100. If this epoch is going well, I will adjust this.<br>\nUntill now, the learning is going well !!</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1058564,
          "author_name": "aliabdin1",
          "author_url": "",
          "post_date": "10/23/2020 21:51:01",
          "content": "<p>I posted some ideas on a different topic:<br>\n<a href=\"https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/187773\" target=\"_blank\">https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/187773</a></p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1057107,
      "author_name": "deepakrajpurushothaman",
      "author_url": "",
      "post_date": "10/22/2020 11:21:13",
      "content": "<p><a href=\"https://www.kaggle.com/youhanlee\" target=\"_blank\">@youhanlee</a> you can refer to part 3 of following post</p>\n<p><a href=\"https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/191599\" target=\"_blank\">https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/191599</a> </p>",
      "votes": null,
      "replies": [
        {
          "id": 1057188,
          "author_name": "youhanlee",
          "author_url": "",
          "post_date": "10/22/2020 12:59:07",
          "content": "<p>I tried <br>\n(1) error = torch.log(confidences + 1e-10) - 0.5 * torch.sum(error, dim=-1)<br>\n(2) assert torch.allclose(torch.sum(confidences, dim=1), confidences.new_ones((batch_size,)), atol=1e-04), \"confidences should sum to 1\" - change atol to 1e04<br>\n(3) gradient clipping<br>\n(4) lower learning rate (1-e5)</p>\n<p>Only (3) works for me until now. But there would be other options to fix this issue.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1078446,
      "author_name": "yousof9",
      "author_url": "",
      "post_date": "11/14/2020 19:53:36",
      "content": "<p><a href=\"https://www.kaggle.com/youhanlee\" target=\"_blank\">@youhanlee</a> still having issues with this? I am clipping the gradients but still experiencing this..</p>",
      "votes": null,
      "replies": [
        {
          "id": 1078455,
          "author_name": "aliabdin1",
          "author_url": "",
          "post_date": "11/14/2020 20:08:55",
          "content": "<p>Did you check:<br>\n<a href=\"https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/187773\" target=\"_blank\">https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/187773</a></p>\n<p>I fixed the problem by implementing the easy \"fixes\"</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1084558,
      "author_name": "keremt",
      "author_url": "",
      "post_date": "11/20/2020 06:37:15",
      "content": "<p>Try using logits with torch.log_softmax instead of probabilities and torch.log.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1056859": "I want to develop my new models, not fixing this bug..",
    "1056969": "Clipping grad norm and adjusting learning rate is working now. \nPlease don't stop until tomorrow..",
    "1057107": "youhanlee you can refer to part 3 of following post\n\nhttps://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/191599",
    "1057147": "What did you clip norms at? I never had this issue, but did occasionally get a massive spike in loss/grad, so I just clipped at 1.",
    "1057168": "I clipped at 100. If this epoch is going well, I will adjust this.\nUntill now, the learning is going well !!",
    "1057188": "I tried \n(1) error = torch.log(confidences + 1e-10) - 0.5 * torch.sum(error, dim=-1)\n(2) assert torch.allclose(torch.sum(confidences, dim=1), confidences.new_ones((batch_size,)), atol=1e-04), \"confidences should sum to 1\" - change atol to 1e04\n(3) gradient clipping\n(4) lower learning rate (1-e5)\n\nOnly (3) works for me until now. But there would be other options to fix this issue.",
    "1058564": "I posted some ideas on a different topic:\nhttps://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/187773",
    "1078446": "youhanlee still having issues with this? I am clipping the gradients but still experiencing this..",
    "1078455": "Did you check:\nhttps://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/187773\n\nI fixed the problem by implementing the easy \"fixes\"",
    "1084558": "Try using logits with torch.log_softmax instead of probabilities and torch.log."
  },
  "source": "meta"
}