{
  "id": 402854,
  "title": "4th place solution",
  "url": "/competitions/icecube-neutrinos-in-deep-ice/discussion/402854",
  "author_name": "",
  "post_date": "2023-04-20T00:41:14.619950900Z",
  "votes": 29,
  "comment_count": 4,
  "views": 0,
  "content": "<h1>TLDR</h1>\n<p>Ensemble of modified transformer models with pulses as tokens.</p>\n<p>Layers: 8-16</p>\n<p>Embed: 128-384</p>\n<p>FF: 1024-2048</p>\n<p>SeqL: 96-256</p>\n<p>Heads: 4-16</p>\n<p>2 prediction heads:</p>\n<ol>\n<li>regression for XYZ vector (MSE)</li>\n<li>classification for locations at fibonacci sphere (smooth BCE)</li>\n</ol>\n<p>2 stage training, main + finetune on longer sequences</p>\n<p>CV-LB scores well corelated, Best single model is about .967/.968, about 30 minutes for whole test.<br>\nIll update this later on</p>",
  "messages": [
    {
      "id": "2227700",
      "postDate": "04/20/2023 00:41:14",
      "content": "<h1>TLDR</h1>\n<p>Ensemble of modified transformer models with pulses as tokens.</p>\n<p>Layers: 8-16</p>\n<p>Embed: 128-384</p>\n<p>FF: 1024-2048</p>\n<p>SeqL: 96-256</p>\n<p>Heads: 4-16</p>\n<p>2 prediction heads:</p>\n<ol>\n<li>regression for XYZ vector (MSE)</li>\n<li>classification for locations at fibonacci sphere (smooth BCE)</li>\n</ol>\n<p>2 stage training, main + finetune on longer sequences</p>\n<p>CV-LB scores well corelated, Best single model is about .967/.968, about 30 minutes for whole test.<br>\nIll update this later on</p>",
      "rawMarkdown": "# TLDR\n\nEnsemble of modified transformer models with pulses as tokens.\n\nLayers: 8-16\n\nEmbed: 128-384\n\nFF: 1024-2048\n\nSeqL: 96-256\n\nHeads: 4-16\n\n2 prediction heads:\n\n1. regression for XYZ vector (MSE)\n2. classification for locations at fibonacci sphere (smooth BCE)\n\n2 stage training, main + finetune on longer sequences\n\nCV-LB scores well corelated, Best single model is about .967/.968, about 30 minutes for whole test.\nIll update this later on",
      "votes": null
    },
    {
      "id": "2227729",
      "postDate": "04/20/2023 01:39:12",
      "content": "<p>Hm, interesting to read. Also tried a custom transformer that predicted XYZ vector with MSE and VmF loss. However, in our case that didn't work well. </p>",
      "rawMarkdown": "Hm, interesting to read. Also tried a custom transformer that predicted XYZ vector with MSE and VmF loss. However, in our case that didn't work well.",
      "votes": null
    },
    {
      "id": "2227730",
      "postDate": "04/20/2023 01:39:33",
      "content": "<p>Awesome! I'm looking forward to seeing your notebooks. How was your training time?</p>",
      "rawMarkdown": "Awesome! I'm looking forward to seeing your notebooks. How was your training time?",
      "votes": null
    },
    {
      "id": "2227778",
      "postDate": "04/20/2023 02:44:33",
      "content": "<p>Impressive work！Looking forward to more details！</p>",
      "rawMarkdown": "Impressive work！Looking forward to more details！",
      "votes": null
    },
    {
      "id": "2227796",
      "postDate": "04/20/2023 03:00:29",
      "content": "<p>Seems like our solutions are quite similar.</p>\n<p>Regression to XYZ with MSE, I tried the same and it was super unstable for me, maybe it was the hyperparameters.</p>",
      "rawMarkdown": "Seems like our solutions are quite similar.\n\nRegression to XYZ with MSE, I tried the same and it was super unstable for me, maybe it was the hyperparameters.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2227729,
      "author_name": "manwithaflower",
      "author_url": "",
      "post_date": "04/20/2023 01:39:12",
      "content": "<p>Hm, interesting to read. Also tried a custom transformer that predicted XYZ vector with MSE and VmF loss. However, in our case that didn't work well. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2227730,
      "author_name": "junseonglee11",
      "author_url": "",
      "post_date": "04/20/2023 01:39:33",
      "content": "<p>Awesome! I'm looking forward to seeing your notebooks. How was your training time?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2227778,
      "author_name": "horikitasaku",
      "author_url": "",
      "post_date": "04/20/2023 02:44:33",
      "content": "<p>Impressive work！Looking forward to more details！</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2227796,
      "author_name": "dipamc77",
      "author_url": "",
      "post_date": "04/20/2023 03:00:29",
      "content": "<p>Seems like our solutions are quite similar.</p>\n<p>Regression to XYZ with MSE, I tried the same and it was super unstable for me, maybe it was the hyperparameters.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2227700": "# TLDR\n\nEnsemble of modified transformer models with pulses as tokens.\n\nLayers: 8-16\n\nEmbed: 128-384\n\nFF: 1024-2048\n\nSeqL: 96-256\n\nHeads: 4-16\n\n2 prediction heads:\n\n1. regression for XYZ vector (MSE)\n2. classification for locations at fibonacci sphere (smooth BCE)\n\n2 stage training, main + finetune on longer sequences\n\nCV-LB scores well corelated, Best single model is about .967/.968, about 30 minutes for whole test.\nIll update this later on",
    "2227729": "Hm, interesting to read. Also tried a custom transformer that predicted XYZ vector with MSE and VmF loss. However, in our case that didn't work well.",
    "2227730": "Awesome! I'm looking forward to seeing your notebooks. How was your training time?",
    "2227778": "Impressive work！Looking forward to more details！",
    "2227796": "Seems like our solutions are quite similar.\n\nRegression to XYZ with MSE, I tried the same and it was super unstable for me, maybe it was the hyperparameters."
  },
  "source": "meta"
}