{
  "id": 399575,
  "title": "Do not take log(number of auxiliary pulses)",
  "url": "/competitions/icecube-neutrinos-in-deep-ice/discussion/399575",
  "author_name": "",
  "post_date": "2023-04-04T16:10:32.005229700Z",
  "votes": -4,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Batch 379 has an event with 0 auxiliary pulses, that will lead to log(0) which will create an NaN which will f*ck your training process :p</p>",
  "messages": [
    {
      "id": "2209334",
      "postDate": "04/04/2023 16:10:32",
      "content": "<p>Batch 379 has an event with 0 auxiliary pulses, that will lead to log(0) which will create an NaN which will f*ck your training process :p</p>",
      "rawMarkdown": "Batch 379 has an event with 0 auxiliary pulses, that will lead to log(0) which will create an NaN which will f*ck your training process :p",
      "votes": null
    },
    {
      "id": "2209484",
      "postDate": "04/04/2023 17:46:47",
      "content": "<p>But you can prevent that to add a small eps to zero…</p>",
      "rawMarkdown": "But you can prevent that to add a small eps to zero...",
      "votes": null
    },
    {
      "id": "2209505",
      "postDate": "04/04/2023 18:05:59",
      "content": "<p>Yep, that's a way to do it :)</p>\n<p>I just wanted to warn people because it's pretty deep into the data and it's something that's easy to overlook. </p>\n<p>For example, if someone is training a model for days, and, 379 is one of the last batches, that might cause a set back with the training.</p>\n<p>Also, if someone doesn't use 379 to train and an event without auxiliary pulses is in the test data, that might cause the submission to become useless.</p>\n<p>In any case, I admit it is a pretty specific situation, I just didn't want anyone to encounter the same thing that I did the hard way, and maybe save some headaches.</p>",
      "rawMarkdown": "Yep, that's a way to do it :)\n\nI just wanted to warn people because it's pretty deep into the data and it's something that's easy to overlook. \n\nFor example, if someone is training a model for days, and, 379 is one of the last batches, that might cause a set back with the training.\n\nAlso, if someone doesn't use 379 to train and an event without auxiliary pulses is in the test data, that might cause the submission to become useless.\n\nIn any case, I admit it is a pretty specific situation, I just didn't want anyone to encounter the same thing that I did the hard way, and maybe save some headaches.",
      "votes": null
    },
    {
      "id": "2221279",
      "postDate": "04/14/2023 05:40:29",
      "content": "<p>Maybe adding a small constant to the number of auxiliary pulses before taking log would help to avoid NaNs and also preserve some information from those events.</p>",
      "rawMarkdown": "Maybe adding a small constant to the number of auxiliary pulses before taking log would help to avoid NaNs and also preserve some information from those events.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2209484,
      "author_name": "killimi",
      "author_url": "",
      "post_date": "04/04/2023 17:46:47",
      "content": "<p>But you can prevent that to add a small eps to zero…</p>",
      "votes": null,
      "replies": [
        {
          "id": 2209505,
          "author_name": "araraonline",
          "author_url": "",
          "post_date": "04/04/2023 18:05:59",
          "content": "<p>Yep, that's a way to do it :)</p>\n<p>I just wanted to warn people because it's pretty deep into the data and it's something that's easy to overlook. </p>\n<p>For example, if someone is training a model for days, and, 379 is one of the last batches, that might cause a set back with the training.</p>\n<p>Also, if someone doesn't use 379 to train and an event without auxiliary pulses is in the test data, that might cause the submission to become useless.</p>\n<p>In any case, I admit it is a pretty specific situation, I just didn't want anyone to encounter the same thing that I did the hard way, and maybe save some headaches.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2221279,
      "author_name": "yus002",
      "author_url": "",
      "post_date": "04/14/2023 05:40:29",
      "content": "<p>Maybe adding a small constant to the number of auxiliary pulses before taking log would help to avoid NaNs and also preserve some information from those events.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2209334": "Batch 379 has an event with 0 auxiliary pulses, that will lead to log(0) which will create an NaN which will f*ck your training process :p",
    "2209484": "But you can prevent that to add a small eps to zero...",
    "2209505": "Yep, that's a way to do it :)\n\nI just wanted to warn people because it's pretty deep into the data and it's something that's easy to overlook. \n\nFor example, if someone is training a model for days, and, 379 is one of the last batches, that might cause a set back with the training.\n\nAlso, if someone doesn't use 379 to train and an event without auxiliary pulses is in the test data, that might cause the submission to become useless.\n\nIn any case, I admit it is a pretty specific situation, I just didn't want anyone to encounter the same thing that I did the hard way, and maybe save some headaches.",
    "2221279": "Maybe adding a small constant to the number of auxiliary pulses before taking log would help to avoid NaNs and also preserve some information from those events."
  },
  "source": "meta"
}