{
  "id": 380601,
  "title": "Notebook takes ~8x longer when submitted",
  "url": "/competitions/icecube-neutrinos-in-deep-ice/discussion/380601",
  "author_name": "Andrew",
  "post_date": "2023-01-23T16:51:31.727000",
  "votes": 12,
  "comment_count": 17,
  "views": 0,
  "content": "<ul>\n<li>In local testing, my prediction method can process 1M events (from the training set) in just over half an hour.</li>\n<li>When I submit the code to the leaderboard, to run over the hidden test set (and not even loading the training set), it takes ~4 hours.</li>\n</ul>\n<p>The <a href=\"https://www.kaggle.com/competitions/icecube-neutrinos-in-deep-ice/data\" target=\"_blank\">Data section</a> says…</p>\n<blockquote>\n  <p>Expect to see roughly one million events in the hidden test set.</p>\n</blockquote>\n<p>…so I would expect it to take about half an hour.</p>\n<ol>\n<li>Is anybody else seeing this?</li>\n<li>Is there something different about the test data?<ul>\n<li>Are there 10x more non-auxiliary samples in each event?</li>\n<li>Are there actually more like 10M events?</li>\n<li>Is the data batched differently?</li></ul></li>\n<li>Is there something different about the way Kaggle runs submitted notebooks?<ul>\n<li>Poorer hardware?</li>\n<li>Large queue of submissions so it takes ages to get started?</li>\n<li>I know they add a little jitter to prevent data leakage by timings, but surely not &gt;3 hours.</li></ul></li>\n</ol>",
  "messages": [
    {
      "id": 2112485,
      "postDate": "2023-01-23T16:51:31.727Z",
      "content": "<ul>\n<li>In local testing, my prediction method can process 1M events (from the training set) in just over half an hour.</li>\n<li>When I submit the code to the leaderboard, to run over the hidden test set (and not even loading the training set), it takes ~4 hours.</li>\n</ul>\n<p>The <a href=\"https://www.kaggle.com/competitions/icecube-neutrinos-in-deep-ice/data\" target=\"_blank\">Data section</a> says…</p>\n<blockquote>\n  <p>Expect to see roughly one million events in the hidden test set.</p>\n</blockquote>\n<p>…so I would expect it to take about half an hour.</p>\n<ol>\n<li>Is anybody else seeing this?</li>\n<li>Is there something different about the test data?<ul>\n<li>Are there 10x more non-auxiliary samples in each event?</li>\n<li>Are there actually more like 10M events?</li>\n<li>Is the data batched differently?</li></ul></li>\n<li>Is there something different about the way Kaggle runs submitted notebooks?<ul>\n<li>Poorer hardware?</li>\n<li>Large queue of submissions so it takes ages to get started?</li>\n<li>I know they add a little jitter to prevent data leakage by timings, but surely not &gt;3 hours.</li></ul></li>\n</ol>",
      "rawMarkdown": "- In local testing, my prediction method can process 1M events (from the training set) in just over half an hour.\n- When I submit the code to the leaderboard, to run over the hidden test set (and not even loading the training set), it takes ~4 hours.\n\nThe [Data section](https://www.kaggle.com/competitions/icecube-neutrinos-in-deep-ice/data) says...\n\n> Expect to see roughly one million events in the hidden test set.\n\n...so I would expect it to take about half an hour.\n\n1. Is anybody else seeing this?\n1. Is there something different about the test data?\n  - Are there 10x more non-auxiliary samples in each event?\n  - Are there actually more like 10M events?\n  - Is the data batched differently?\n1. Is there something different about the way Kaggle runs submitted notebooks?\n  - Poorer hardware?\n  - Large queue of submissions so it takes ages to get started?\n  - I know they add a little jitter to prevent data leakage by timings, but surely not >3 hours.",
      "votes": 11
    },
    {
      "id": 2114392,
      "postDate": "2023-01-25T00:31:24.003Z",
      "content": "<p>My results on my 1.214 sub:<br>\nTrain: 0:00:18 per batch (200k)<br>\nTest: right about 0:02:00 flat. So seems about right for 1 million events.</p>\n<p>Sorry, couldn't resist using this thread as a brag post. Tacky, I know!<br>\nBut I'll post it publicly tonight, I hope, so think of this as a teaser trailer :)</p>",
      "rawMarkdown": "My results on my 1.214 sub:\nTrain: 0:00:18 per batch (200k)\nTest: right about 0:02:00 flat. So seems about right for 1 million events.\n\nSorry, couldn't resist using this thread as a brag post. Tacky, I know!\nBut I'll post it publicly tonight, I hope, so think of this as a teaser trailer :)",
      "votes": 3,
      "replies": [
        {
          "id": 2116990,
          "postDate": "2023-01-26T21:24:49.683Z",
          "content": "<p>Excited to see!!</p>",
          "rawMarkdown": "Excited to see!!",
          "votes": 1,
          "replies": [
            {
              "id": 2119806,
              "postDate": "2023-01-29T05:48:52.350Z",
              "content": "<p><a href=\"https://www.kaggle.com/dschettler8845\" target=\"_blank\">@dschettler8845</a> It's up now!</p>",
              "rawMarkdown": "@dschettler8845 It's up now!"
            }
          ]
        }
      ]
    },
    {
      "id": 2122828,
      "postDate": "2023-01-31T06:44:27.597Z",
      "content": "<p>Strange result, first time I've seen this.   I've seen a number of (purposefully) failed subs, but they all come back immediately.</p>\n<p>This one appears to still be processing, 34 minutes after the failure.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9052057%2Fe3e6579d26226f0ec3cf3a3b392e316f%2Ffailed.png?generation=1675147328276327&amp;alt=media\" alt=\"\"></p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9052057%2F51da0b5ea05e8e19d33822730e56e34d%2Fspinning.png?generation=1675147445665854&amp;alt=media\" alt=\"\"></p>\n<p>Note that there was a syntax error in the notebook. <a href=\"https://www.kaggle.com/code/kaggleqrdl/probing-notebook?scriptVersionId=117791619\" target=\"_blank\">https://www.kaggle.com/code/kaggleqrdl/probing-notebook?scriptVersionId=117791619</a></p>",
      "rawMarkdown": "Strange result, first time I've seen this.   I've seen a number of (purposefully) failed subs, but they all come back immediately.\n\nThis one appears to still be processing, 34 minutes after the failure.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9052057%2Fe3e6579d26226f0ec3cf3a3b392e316f%2Ffailed.png?generation=1675147328276327&alt=media)\n\n\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9052057%2F51da0b5ea05e8e19d33822730e56e34d%2Fspinning.png?generation=1675147445665854&alt=media)\n\nNote that there was a syntax error in the notebook. https://www.kaggle.com/code/kaggleqrdl/probing-notebook?scriptVersionId=117791619",
      "replies": [
        {
          "id": 2123992,
          "postDate": "2023-01-31T19:27:17.160Z",
          "content": "<p>Still going.  :)</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9052057%2F540fde32185433d98dfd35f0cab48780%2Fstillgoing.png?generation=1675193235870967&amp;alt=media\" alt=\"\"></p>",
          "rawMarkdown": "Still going.  :)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9052057%2F540fde32185433d98dfd35f0cab48780%2Fstillgoing.png?generation=1675193235870967&alt=media)"
        }
      ]
    },
    {
      "id": 2112958,
      "postDate": "2023-01-24T01:11:23.810Z",
      "content": "<p>It looks like there's a queue to get scored - my subs finish long before they're scored fwiw.</p>\n<p>On the subject of hardware…what's fair to assume re: the ram provisioned for the actual test data run?  </p>",
      "rawMarkdown": "It looks like there's a queue to get scored - my subs finish long before they're scored fwiw.\n\nOn the subject of hardware...what's fair to assume re: the ram provisioned for the actual test data run?  "
    },
    {
      "id": 2112822,
      "postDate": "2023-01-23T21:32:12.463Z",
      "content": "<p>Should be easy to probe for this info.   just fail the submit </p>\n<p>..  ok creating a probing notebook which just zeros out the angles, but tests invariants for success/failure.  will share</p>\n<p>Thinking about it, the delay actually might not be your code VM but rather the pool evaluating the submissions is backed up.  Possibly they didn't think to provision it appropriately.</p>",
      "rawMarkdown": "Should be easy to probe for this info.   just fail the submit \n\n..  ok creating a probing notebook which just zeros out the angles, but tests invariants for success/failure.  will share\n\nThinking about it, the delay actually might not be your code VM but rather the pool evaluating the submissions is backed up.  Possibly they didn't think to provision it appropriately.\n",
      "replies": [
        {
          "id": 2112852,
          "postDate": "2023-01-23T22:21:42.947Z",
          "content": "<p>fwiw, my zero angle submit returned in about 2 minutes, strangely it beat the reference sub with 1.533</p>\n<p>There is somewhere between 900k and 1100k events.  This submit succeeds:</p>\n<pre><code> numpy  np \n pandas  pd \n\ntests = pd.read_parquet()\n\n (tests) &lt;   (tests) &gt; :  \n     Exception\n\n\n tqdm\nevents = tests.event_id\nres = []\n (,)  f:\n    f.write()\n     i  tqdm.tqdm(events):\n        f.write() \n</code></pre>\n<p>I'm all out of quota for today, but I'll add a bunch of conditions over the next week or so.  Let me know if there's anything you'd like me to test.</p>\n<p>eg:</p>\n<ul>\n<li>number of batches</li>\n<li>max/min values for various columns</li>\n<li>duplicate counts</li>\n<li>nan counts</li>\n<li>statistical properties, like std/mean, ?<br>\netc.. </li>\n</ul>\n<p>In general we should see a roughly similar distribution between test and train, which is what we want to test for.</p>",
          "rawMarkdown": "fwiw, my zero angle submit returned in about 2 minutes, strangely it beat the reference sub with 1.533\n\nThere is somewhere between 900k and 1100k events.  This submit succeeds:\n\n\n```python\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\ntests = pd.read_parquet('/kaggle/input/icecube-neutrinos-in-deep-ice/test_meta.parquet')\n\nif len(tests) < 900000 or len(tests) > 1100000:  \n    raise Exception\n\n\nimport tqdm\nevents = tests.event_id\nres = []\nwith open(\"submission.csv\",\"a\") as f:\n    f.write(\"event_id,azimuth,zenith\\n\")\n    for i in tqdm.tqdm(events):\n        f.write(f\"{i},0,0\\n\") \n```\n\nI'm all out of quota for today, but I'll add a bunch of conditions over the next week or so.  Let me know if there's anything you'd like me to test.\n\neg:\n- number of batches\n- max/min values for various columns\n- duplicate counts\n- nan counts\n- statistical properties, like std/mean, ?\netc.. \n\nIn general we should see a roughly similar distribution between test and train, which is what we want to test for.",
          "votes": 5,
          "replies": [
            {
              "id": 2112883,
              "postDate": "2023-01-23T22:56:04.153Z",
              "content": "<p>Thanks for using your subs and sharing with us. <br>\nIt will be interesting to know min/max number of pulses for aux/non aux samples</p>\n<p>Ps<br>\nBtw i can uderstand the logic to probe binary outcomes but i'm missing the setup to probe eg stats like min,max etc </p>",
              "rawMarkdown": "Thanks for using your subs and sharing with us. \nIt will be interesting to know min/max number of pulses for aux/non aux samples\n\nPs\nBtw i can uderstand the logic to probe binary outcomes but i'm missing the setup to probe eg stats like min,max etc "
            },
            {
              "id": 2112888,
              "postDate": "2023-01-23T23:08:02.307Z",
              "content": "<p>One basic technique is probing values into buckets via different scores (eg, if std falls in this bucket, use these angles which gives the appropriate score).  Furthermore, it's not too difficult to think of ways to have quite a lot of buckets, which allows for fairly granular probing.  Once I have some unique  submission scores I'll add the bucket support to the notebook.  </p>\n<p>There's actually some fairly limitless probing that you can do if you put your mind to it.  I'm not really a huge fan of the LB for that reason, because it's an Oracle that folks can and very likely do abuse.</p>\n<p>I wish Kaggle could find someway to move beyond the public LB, at least as is. The current GoDaddy comp is a potential solution.  Have a public LB, but reveal the data in stages such that probing gets no one really anywhere.  Some public LB data at the end is fine as long as the number of potential probes after that is quite small.</p>",
              "rawMarkdown": "One basic technique is probing values into buckets via different scores (eg, if std falls in this bucket, use these angles which gives the appropriate score).  Furthermore, it's not too difficult to think of ways to have quite a lot of buckets, which allows for fairly granular probing.  Once I have some unique  submission scores I'll add the bucket support to the notebook.  \n\nThere's actually some fairly limitless probing that you can do if you put your mind to it.  I'm not really a huge fan of the LB for that reason, because it's an Oracle that folks can and very likely do abuse.\n\nI wish Kaggle could find someway to move beyond the public LB, at least as is. The current GoDaddy comp is a potential solution.  Have a public LB, but reveal the data in stages such that probing gets no one really anywhere.  Some public LB data at the end is fine as long as the number of potential probes after that is quite small."
            },
            {
              "id": 2112910,
              "postDate": "2023-01-23T23:59:53.553Z",
              "rawMarkdown": "",
              "isDeleted": true
            },
            {
              "id": 2113168,
              "postDate": "2023-01-24T05:56:13.427Z",
              "content": "<p>If it's 1million events, but taking way longer, there could be more outliers with thousands of points in a single event than in the sub-section of train use for comparison. But it is strange. </p>",
              "rawMarkdown": "If it's 1million events, but taking way longer, there could be more outliers with thousands of points in a single event than in the sub-section of train use for comparison. But it is strange. "
            },
            {
              "id": 2113717,
              "postDate": "2023-01-24T12:54:59.323Z",
              "content": "<p>Ok shared it here.  <a href=\"https://www.kaggle.com/code/kaggleqrdl/probing-notebook\" target=\"_blank\">https://www.kaggle.com/code/kaggleqrdl/probing-notebook</a>  check back every day for new and exciting results!  =)</p>",
              "rawMarkdown": "Ok shared it here.  https://www.kaggle.com/code/kaggleqrdl/probing-notebook  check back every day for new and exciting results!  =)"
            },
            {
              "id": 2113730,
              "postDate": "2023-01-24T13:07:25.970Z",
              "content": "<p>I had one of my submissions fail due to hitting the 9 hour mark.</p>\n<p>The code itself ran successfully in ~6 minutes on submission, but didn't score and then failed at the 9 hour mark. It also included a while loop to modify sensor data sampling.</p>\n<p>Tldr: I now think this delay is the 1million test obs running.</p>",
              "rawMarkdown": "I had one of my submissions fail due to hitting the 9 hour mark.\n\nThe code itself ran successfully in ~6 minutes on submission, but didn't score and then failed at the 9 hour mark. It also included a while loop to modify sensor data sampling.\n\nTldr: I now think this delay is the 1million test obs running."
            },
            {
              "id": 2113736,
              "postDate": "2023-01-24T13:18:51.957Z",
              "content": "<p>fwiw, I've submitted 10 times now and have yet to see this issue.  This is not to say the problem doesn't exist, just that I haven't encountered it.</p>",
              "rawMarkdown": "fwiw, I've submitted 10 times now and have yet to see this issue.  This is not to say the problem doesn't exist, just that I haven't encountered it.",
              "votes": 1
            }
          ]
        }
      ]
    },
    {
      "id": 2112698,
      "postDate": "2023-01-23T19:06:15.810Z",
      "content": "<p>I am having exactly the same problem. When I run my notebook on the train dataset it should take ~20min for 1M events, however I have sumbitted it 1 hour ago and it has still not finished…</p>",
      "rawMarkdown": "I am having exactly the same problem. When I run my notebook on the train dataset it should take ~20min for 1M events, however I have sumbitted it 1 hour ago and it has still not finished..."
    },
    {
      "id": 2112562,
      "postDate": "2023-01-23T17:27:17.373Z",
      "content": "<p>My prediction method (<a href=\"https://www.kaggle.com/code/shlomoron/icecube-eda-pca-baseline-cv-1-28-lb-1-274\" target=\"_blank\">here</a>) run on two batches (I think it is 400K events, right?) in about half an hour. I did not check how much time it took when I submitted it, but I think it is something around three hours. (Maybe someone from the ~30 people who already submitted it too can give an exact number).<br>\nSo…<br>\nOn the one hand, yes, 400K = 0.5h in validation vs. 1M = ~3h is quite the discrepancy.<br>\nOn the other hand…in your case, it was 1M=0.5h versus 1M = 4h. So much worse than my case.<br>\nTry to run on the first two batches and compare to my numbers; maybe there is a difference in the number of pulses from batch to batch.</p>",
      "rawMarkdown": "My prediction method ([here](https://www.kaggle.com/code/shlomoron/icecube-eda-pca-baseline-cv-1-28-lb-1-274)) run on two batches (I think it is 400K events, right?) in about half an hour. I did not check how much time it took when I submitted it, but I think it is something around three hours. (Maybe someone from the ~30 people who already submitted it too can give an exact number).\nSo...\nOn the one hand, yes, 400K = 0.5h in validation vs. 1M = ~3h is quite the discrepancy.\nOn the other hand...in your case, it was 1M=0.5h versus 1M = 4h. So much worse than my case.\nTry to run on the first two batches and compare to my numbers; maybe there is a difference in the number of pulses from batch to batch."
    }
  ],
  "comments": [
    {
      "id": 2114392,
      "author_name": "Robert Hatch",
      "author_url": "",
      "post_date": "2023-01-25T00:31:24.003000",
      "content": "<p>My results on my 1.214 sub:<br>\nTrain: 0:00:18 per batch (200k)<br>\nTest: right about 0:02:00 flat. So seems about right for 1 million events.</p>\n<p>Sorry, couldn't resist using this thread as a brag post. Tacky, I know!<br>\nBut I'll post it publicly tonight, I hope, so think of this as a teaser trailer :)</p>",
      "votes": 3,
      "replies": [
        {
          "id": 2116990,
          "author_name": "Darien Schettler",
          "author_url": "",
          "post_date": "2023-01-26T21:24:49.683000",
          "content": "<p>Excited to see!!</p>",
          "votes": 1,
          "replies": [
            {
              "id": 2119806,
              "author_name": "Robert Hatch",
              "author_url": "",
              "post_date": "2023-01-29T05:48:52.350000",
              "content": "<p><a href=\"https://www.kaggle.com/dschettler8845\" target=\"_blank\">@dschettler8845</a> It's up now!</p>",
              "votes": 0,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 2122828,
      "author_name": "@kaggleqrdl",
      "author_url": "",
      "post_date": "2023-01-31T06:44:27.597000",
      "content": "<p>Strange result, first time I've seen this.   I've seen a number of (purposefully) failed subs, but they all come back immediately.</p>\n<p>This one appears to still be processing, 34 minutes after the failure.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9052057%2Fe3e6579d26226f0ec3cf3a3b392e316f%2Ffailed.png?generation=1675147328276327&amp;alt=media\" alt=\"\"></p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9052057%2F51da0b5ea05e8e19d33822730e56e34d%2Fspinning.png?generation=1675147445665854&amp;alt=media\" alt=\"\"></p>\n<p>Note that there was a syntax error in the notebook. <a href=\"https://www.kaggle.com/code/kaggleqrdl/probing-notebook?scriptVersionId=117791619\" target=\"_blank\">https://www.kaggle.com/code/kaggleqrdl/probing-notebook?scriptVersionId=117791619</a></p>",
      "votes": 0,
      "replies": [
        {
          "id": 2123992,
          "author_name": "@kaggleqrdl",
          "author_url": "",
          "post_date": "2023-01-31T19:27:17.160000",
          "content": "<p>Still going.  :)</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9052057%2F540fde32185433d98dfd35f0cab48780%2Fstillgoing.png?generation=1675193235870967&amp;alt=media\" alt=\"\"></p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 2112958,
      "author_name": "Sean McManus",
      "author_url": "",
      "post_date": "2023-01-24T01:11:23.810000",
      "content": "<p>It looks like there's a queue to get scored - my subs finish long before they're scored fwiw.</p>\n<p>On the subject of hardware…what's fair to assume re: the ram provisioned for the actual test data run?  </p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 2112822,
      "author_name": "@kaggleqrdl",
      "author_url": "",
      "post_date": "2023-01-23T21:32:12.463000",
      "content": "<p>Should be easy to probe for this info.   just fail the submit </p>\n<p>..  ok creating a probing notebook which just zeros out the angles, but tests invariants for success/failure.  will share</p>\n<p>Thinking about it, the delay actually might not be your code VM but rather the pool evaluating the submissions is backed up.  Possibly they didn't think to provision it appropriately.</p>",
      "votes": 0,
      "replies": [
        {
          "id": 2112852,
          "author_name": "@kaggleqrdl",
          "author_url": "",
          "post_date": "2023-01-23T22:21:42.947000",
          "content": "<p>fwiw, my zero angle submit returned in about 2 minutes, strangely it beat the reference sub with 1.533</p>\n<p>There is somewhere between 900k and 1100k events.  This submit succeeds:</p>\n<pre><code> numpy  np \n pandas  pd \n\ntests = pd.read_parquet()\n\n (tests) &lt;   (tests) &gt; :  \n     Exception\n\n\n tqdm\nevents = tests.event_id\nres = []\n (,)  f:\n    f.write()\n     i  tqdm.tqdm(events):\n        f.write() \n</code></pre>\n<p>I'm all out of quota for today, but I'll add a bunch of conditions over the next week or so.  Let me know if there's anything you'd like me to test.</p>\n<p>eg:</p>\n<ul>\n<li>number of batches</li>\n<li>max/min values for various columns</li>\n<li>duplicate counts</li>\n<li>nan counts</li>\n<li>statistical properties, like std/mean, ?<br>\netc.. </li>\n</ul>\n<p>In general we should see a roughly similar distribution between test and train, which is what we want to test for.</p>",
          "votes": 5,
          "replies": [
            {
              "id": 2112883,
              "author_name": "Ioannis M",
              "author_url": "",
              "post_date": "2023-01-23T22:56:04.153000",
              "content": "<p>Thanks for using your subs and sharing with us. <br>\nIt will be interesting to know min/max number of pulses for aux/non aux samples</p>\n<p>Ps<br>\nBtw i can uderstand the logic to probe binary outcomes but i'm missing the setup to probe eg stats like min,max etc </p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 2112888,
              "author_name": "@kaggleqrdl",
              "author_url": "",
              "post_date": "2023-01-23T23:08:02.307000",
              "content": "<p>One basic technique is probing values into buckets via different scores (eg, if std falls in this bucket, use these angles which gives the appropriate score).  Furthermore, it's not too difficult to think of ways to have quite a lot of buckets, which allows for fairly granular probing.  Once I have some unique  submission scores I'll add the bucket support to the notebook.  </p>\n<p>There's actually some fairly limitless probing that you can do if you put your mind to it.  I'm not really a huge fan of the LB for that reason, because it's an Oracle that folks can and very likely do abuse.</p>\n<p>I wish Kaggle could find someway to move beyond the public LB, at least as is. The current GoDaddy comp is a potential solution.  Have a public LB, but reveal the data in stages such that probing gets no one really anywhere.  Some public LB data at the end is fine as long as the number of potential probes after that is quite small.</p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 2112910,
              "author_name": "",
              "author_url": "",
              "post_date": "2023-01-23T23:59:53.553000",
              "content": "",
              "votes": 0,
              "replies": []
            },
            {
              "id": 2113168,
              "author_name": "Robert Hatch",
              "author_url": "",
              "post_date": "2023-01-24T05:56:13.427000",
              "content": "<p>If it's 1million events, but taking way longer, there could be more outliers with thousands of points in a single event than in the sub-section of train use for comparison. But it is strange. </p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 2113717,
              "author_name": "@kaggleqrdl",
              "author_url": "",
              "post_date": "2023-01-24T12:54:59.323000",
              "content": "<p>Ok shared it here.  <a href=\"https://www.kaggle.com/code/kaggleqrdl/probing-notebook\" target=\"_blank\">https://www.kaggle.com/code/kaggleqrdl/probing-notebook</a>  check back every day for new and exciting results!  =)</p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 2113730,
              "author_name": "Sean McManus",
              "author_url": "",
              "post_date": "2023-01-24T13:07:25.970000",
              "content": "<p>I had one of my submissions fail due to hitting the 9 hour mark.</p>\n<p>The code itself ran successfully in ~6 minutes on submission, but didn't score and then failed at the 9 hour mark. It also included a while loop to modify sensor data sampling.</p>\n<p>Tldr: I now think this delay is the 1million test obs running.</p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 2113736,
              "author_name": "@kaggleqrdl",
              "author_url": "",
              "post_date": "2023-01-24T13:18:51.957000",
              "content": "<p>fwiw, I've submitted 10 times now and have yet to see this issue.  This is not to say the problem doesn't exist, just that I haven't encountered it.</p>",
              "votes": 1,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 2112698,
      "author_name": "Thomas Leplumey",
      "author_url": "",
      "post_date": "2023-01-23T19:06:15.810000",
      "content": "<p>I am having exactly the same problem. When I run my notebook on the train dataset it should take ~20min for 1M events, however I have sumbitted it 1 hour ago and it has still not finished…</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 2112562,
      "author_name": "greySnow",
      "author_url": "",
      "post_date": "2023-01-23T17:27:17.373000",
      "content": "<p>My prediction method (<a href=\"https://www.kaggle.com/code/shlomoron/icecube-eda-pca-baseline-cv-1-28-lb-1-274\" target=\"_blank\">here</a>) run on two batches (I think it is 400K events, right?) in about half an hour. I did not check how much time it took when I submitted it, but I think it is something around three hours. (Maybe someone from the ~30 people who already submitted it too can give an exact number).<br>\nSo…<br>\nOn the one hand, yes, 400K = 0.5h in validation vs. 1M = ~3h is quite the discrepancy.<br>\nOn the other hand…in your case, it was 1M=0.5h versus 1M = 4h. So much worse than my case.<br>\nTry to run on the first two batches and compare to my numbers; maybe there is a difference in the number of pulses from batch to batch.</p>",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2112485": "- In local testing, my prediction method can process 1M events (from the training set) in just over half an hour.\n- When I submit the code to the leaderboard, to run over the hidden test set (and not even loading the training set), it takes ~4 hours.\n\nThe [Data section](https://www.kaggle.com/competitions/icecube-neutrinos-in-deep-ice/data) says...\n\n> Expect to see roughly one million events in the hidden test set.\n\n...so I would expect it to take about half an hour.\n\n1. Is anybody else seeing this?\n1. Is there something different about the test data?\n  - Are there 10x more non-auxiliary samples in each event?\n  - Are there actually more like 10M events?\n  - Is the data batched differently?\n1. Is there something different about the way Kaggle runs submitted notebooks?\n  - Poorer hardware?\n  - Large queue of submissions so it takes ages to get started?\n  - I know they add a little jitter to prevent data leakage by timings, but surely not >3 hours.",
    "2114392": "My results on my 1.214 sub:\nTrain: 0:00:18 per batch (200k)\nTest: right about 0:02:00 flat. So seems about right for 1 million events.\n\nSorry, couldn't resist using this thread as a brag post. Tacky, I know!\nBut I'll post it publicly tonight, I hope, so think of this as a teaser trailer :)",
    "2122828": "Strange result, first time I've seen this.   I've seen a number of (purposefully) failed subs, but they all come back immediately.\n\nThis one appears to still be processing, 34 minutes after the failure.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9052057%2Fe3e6579d26226f0ec3cf3a3b392e316f%2Ffailed.png?generation=1675147328276327&alt=media)\n\n\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9052057%2F51da0b5ea05e8e19d33822730e56e34d%2Fspinning.png?generation=1675147445665854&alt=media)\n\nNote that there was a syntax error in the notebook. https://www.kaggle.com/code/kaggleqrdl/probing-notebook?scriptVersionId=117791619",
    "2112958": "It looks like there's a queue to get scored - my subs finish long before they're scored fwiw.\n\nOn the subject of hardware...what's fair to assume re: the ram provisioned for the actual test data run?  ",
    "2112822": "Should be easy to probe for this info.   just fail the submit \n\n..  ok creating a probing notebook which just zeros out the angles, but tests invariants for success/failure.  will share\n\nThinking about it, the delay actually might not be your code VM but rather the pool evaluating the submissions is backed up.  Possibly they didn't think to provision it appropriately.\n",
    "2112698": "I am having exactly the same problem. When I run my notebook on the train dataset it should take ~20min for 1M events, however I have sumbitted it 1 hour ago and it has still not finished...",
    "2112562": "My prediction method ([here](https://www.kaggle.com/code/shlomoron/icecube-eda-pca-baseline-cv-1-28-lb-1-274)) run on two batches (I think it is 400K events, right?) in about half an hour. I did not check how much time it took when I submitted it, but I think it is something around three hours. (Maybe someone from the ~30 people who already submitted it too can give an exact number).\nSo...\nOn the one hand, yes, 400K = 0.5h in validation vs. 1M = ~3h is quite the discrepancy.\nOn the other hand...in your case, it was 1M=0.5h versus 1M = 4h. So much worse than my case.\nTry to run on the first two batches and compare to my numbers; maybe there is a difference in the number of pulses from batch to batch."
  }
}