{
  "id": 76563,
  "title": "Commiting via scripts failed ( used almost all available (5368709120) disk space. Disk writes might have failed due to insufficient space.)",
  "url": "/competitions/quora-insincere-questions-classification/discussion/76563",
  "author_name": "",
  "post_date": "2019-01-04T08:21:30.109598300Z",
  "votes": null,
  "comment_count": 13,
  "views": 0,
  "content": "<p>Hi Guys,</p>\n\n<p>I was able to commit everything perfectly. Today i am experiencing a very weird error. Committing failed with following error message. What does this mean? Can anyone be of some help?\n<code>\nFailure Message\nThe kernel used almost all available (5368709120) disk space. Disk writes might have failed due to insufficient space.\n</code></p>",
  "messages": [
    {
      "id": "450060",
      "postDate": "01/04/2019 08:21:30",
      "content": "<p>Hi Guys,</p>\n\n<p>I was able to commit everything perfectly. Today i am experiencing a very weird error. Committing failed with following error message. What does this mean? Can anyone be of some help?\n<code>\nFailure Message\nThe kernel used almost all available (5368709120) disk space. Disk writes might have failed due to insufficient space.\n</code></p>",
      "rawMarkdown": "Hi Guys,\n\nI was able to commit everything perfectly. Today i am experiencing a very weird error. Committing failed with following error message. What does this mean? Can anyone be of some help?\n```\nFailure Message\nThe kernel used almost all available (5368709120) disk space. Disk writes might have failed due to insufficient space.\n```",
      "votes": null
    },
    {
      "id": "450063",
      "postDate": "01/04/2019 08:27:48",
      "content": "<p>doesnt seem like a weird error to me. it seems the message is self-explanatory. isnt it? ;)</p>",
      "rawMarkdown": "doesnt seem like a weird error to me. it seems the message is self-explanatory. isnt it? ;)",
      "votes": null
    },
    {
      "id": "450065",
      "postDate": "01/04/2019 08:31:35",
      "content": "<p>Hi Abhishek, Yes it is . But 5368709120 bytes is nearly 5GB right. And I am  writing only 2 models   to disk, which I make sure, to remove after every epoch using <code>shutil.rmtree</code>. Only the final submission file is what I am writing permanently .</p>",
      "rawMarkdown": "Hi Abhishek, Yes it is . But 5368709120 bytes is nearly 5GB right. And I am  writing only 2 models   to disk, which I make sure, to remove after every epoch using ```shutil.rmtree```. Only the final submission file is what I am writing permanently .",
      "votes": null
    },
    {
      "id": "450110",
      "postDate": "01/04/2019 10:20:17",
      "content": "<p>All the kernels, which were working before is now failing with same error. Any idea?</p>",
      "rawMarkdown": "All the kernels, which were working before is now failing with same error. Any idea?",
      "votes": null
    },
    {
      "id": "450118",
      "postDate": "01/04/2019 10:32:08",
      "content": "<p>Each kernel has individual disk space right????</p>",
      "rawMarkdown": "Each kernel has individual disk space right????",
      "votes": null
    },
    {
      "id": "450220",
      "postDate": "01/04/2019 14:13:06",
      "content": "<p>Here are some findings, if someone could help that will be great. Before going to these, i want to tell, this was working fine, but from today morning, I got into the above mentioned issues . I am attaching the screenshots</p>\n\n<h2>I.) <strong>Before model is trained</strong></h2>\n\n<p><code>1.png</code> shows the space available in <code>/kaggle/working</code></p>\n\n<p><code>2.png</code> shows the files available in <code>/kaggle/working</code></p>\n\n<p><code>3.png</code> shows the size of all files and sub directories available in <code>/kaggle/working</code> .</p>\n\n<h2>II.) <strong>After model trained for epoch 1   ( No models have been saved yet, I consciously checked it)</strong></h2>\n\n<p><code>4.png</code> shows the size of  <code>/kaggle/working</code> is <code>3.7 GB</code>. But how? I am not saving anything yet.</p>\n\n<p><code>5.png</code> shows the files available in <code>/kaggle/working</code> . No new files, same as <code>2.png</code></p>\n\n<p><code>6.png</code> is the output of <code>df -h</code>. From <code>1.png</code>, the filesystem of <code>/kaggle/working</code> is <code>/dev/loop1</code>. Now, here <code>/dev/loop1</code> is mounted on <code>/tmp</code>, and there is no <code>/kaggle/working</code>. And the size of <code>/tmp</code> is exactly same as the size of <code>/kaggle/working</code> from <code>4.png</code>. </p>\n\n<p>Finally,  <code>7.png</code> is showing the individual files and folders in <code>/kaggle</code>. Here <code>/kaggle/working</code> is present and size is just 12KB. Then how come, this disk is full error is coming.</p>\n\n<p>I know, the above is quite confusing, but this is the best way I could explain.</p>",
      "rawMarkdown": "Here are some findings, if someone could help that will be great. Before going to these, i want to tell, this was working fine, but from today morning, I got into the above mentioned issues . I am attaching the screenshots\n\nI.) **Before model is trained**\n----------------------------------------------------------------------------------------------------------------------------\n ```1.png``` shows the space available in ```/kaggle/working```\n\n ```2.png``` shows the files available in ```/kaggle/working```\n\n ```3.png``` shows the size of all files and sub directories available in ```/kaggle/working``` .\n\n\nII.) **After model trained for epoch 1   ( No models have been saved yet, I consciously checked it)**\n----------------------------------------------------------------------------------------------------------------------------\n ```4.png``` shows the size of  ```/kaggle/working``` is ```3.7 GB```. But how? I am not saving anything yet.\n\n ```5.png``` shows the files available in ```/kaggle/working``` . No new files, same as ```2.png```\n\n ```6.png``` is the output of ```df -h```. From ```1.png```, the filesystem of ```/kaggle/working``` is ```/dev/loop1```. Now, here ```/dev/loop1``` is mounted on ```/tmp```, and there is no ```/kaggle/working```. And the size of ```/tmp``` is exactly same as the size of ```/kaggle/working``` from ```4.png```. \n\n Finally,  ```7.png``` is showing the individual files and folders in ```/kaggle```. Here ```/kaggle/working``` is present and size is just 12KB. Then how come, this disk is full error is coming.\n\nI know, the above is quite confusing, but this is the best way I could explain.",
      "votes": null
    },
    {
      "id": "450239",
      "postDate": "01/04/2019 14:36:34",
      "content": "<p>Cannot find any clues from : <a href=\"https://www.kaggle.com/docs/kernels\">https://www.kaggle.com/docs/kernels</a>\nOnly the last line may give some hints : A kernel may write at most 5 Gigabytes of data to disk (as output)</p>\n\n<p>Since you already remove your saved models every epoch, so I am not sure if this issue is a bug. Perhaps you should go to ask in a general forum with staffs. By the way, how big your two saved models are (before removing) ?</p>",
      "rawMarkdown": "Cannot find any clues from : https://www.kaggle.com/docs/kernels\nOnly the last line may give some hints : A kernel may write at most 5 Gigabytes of data to disk (as output)\n\nSince you already remove your saved models every epoch, so I am not sure if this issue is a bug. Perhaps you should go to ask in a general forum with staffs. By the way, how big your two saved models are (before removing) ?",
      "votes": null
    },
    {
      "id": "450241",
      "postDate": "01/04/2019 14:40:51",
      "content": "<p>My model size is <code>449 MB</code> . The weird thing, once I finished an epoch, before saving the model, i checked the disk space. it has consumed <code>3.7 GB / 4.9 GB</code> . But, where does that comes from? I have asked for a general support, but they will take minimum a day, so I think, my experiments are on hold. :-(</p>",
      "rawMarkdown": "My model size is ```449 MB``` . The weird thing, once I finished an epoch, before saving the model, i checked the disk space. it has consumed ```3.7 GB / 4.9 GB``` . But, where does that comes from? I have asked for a general support, but they will take minimum a day, so I think, my experiments are on hold. :-(",
      "votes": null
    },
    {
      "id": "450247",
      "postDate": "01/04/2019 14:50:10",
      "content": "<p>Okay, at least it is great that a professional will take a look at your issue. Grab some coffee or take some other experiments while waiting for some finding ;)</p>",
      "rawMarkdown": "Okay, at least it is great that a professional will take a look at your issue. Grab some coffee or take some other experiments while waiting for some finding ;)",
      "votes": null
    },
    {
      "id": "451767",
      "postDate": "01/07/2019 16:23:35",
      "content": "<p>Hi Sarath, any update on this ? ;) <a href=\"/s4sarath\">@s4sarath</a></p>",
      "rawMarkdown": "Hi Sarath, any update on this ? ;) @s4sarath",
      "votes": null
    },
    {
      "id": "452007",
      "postDate": "01/08/2019 03:01:12",
      "content": "<p>Not yet. They asked me to sent the details 2 days back. I have provided this discussion link, as there is enough screenshots. Anyway, I found an alternative to save and restore, otherwise I would have been in a trouble?</p>",
      "rawMarkdown": "Not yet. They asked me to sent the details 2 days back. I have provided this discussion link, as there is enough screenshots. Anyway, I found an alternative to save and restore, otherwise I would have been in a trouble?",
      "votes": null
    },
    {
      "id": "452027",
      "postDate": "01/08/2019 03:57:59",
      "content": "<p>Okay, that means you can continue to experiment as you wanted. That should be enough! and it looks like your idea worked quite well too, good job!</p>",
      "rawMarkdown": "Okay, that means you can continue to experiment as you wanted. That should be enough! and it looks like your idea worked quite well too, good job!",
      "votes": null
    },
    {
      "id": "452070",
      "postDate": "01/08/2019 06:29:11",
      "content": "<p>Sort of, but I think it overfits though. Lets see.</p>",
      "rawMarkdown": "Sort of, but I think it overfits though. Lets see.",
      "votes": null
    },
    {
      "id": "457690",
      "postDate": "01/18/2019 00:03:51",
      "content": "<p>If I follow well:</p>\n\n<p><code>4.png</code> shows <code>3.7 GB</code> used on the filesystem <code>/kaggle/working</code> is part of (not that that directory contains that usage).</p>\n\n<p>Look at what goes in <code>/tmp</code> before and after the model training.</p>",
      "rawMarkdown": "If I follow well:\n\n`4.png` shows `3.7 GB` used on the filesystem `/kaggle/working` is part of (not that that directory contains that usage).\n\nLook at what goes in `/tmp` before and after the model training.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 450063,
      "author_name": "abhishek",
      "author_url": "",
      "post_date": "01/04/2019 08:27:48",
      "content": "<p>doesnt seem like a weird error to me. it seems the message is self-explanatory. isnt it? ;)</p>",
      "votes": null,
      "replies": [
        {
          "id": 450065,
          "author_name": "s4sarath",
          "author_url": "",
          "post_date": "01/04/2019 08:31:35",
          "content": "<p>Hi Abhishek, Yes it is . But 5368709120 bytes is nearly 5GB right. And I am  writing only 2 models   to disk, which I make sure, to remove after every epoch using <code>shutil.rmtree</code>. Only the final submission file is what I am writing permanently .</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 450110,
      "author_name": "s4sarath",
      "author_url": "",
      "post_date": "01/04/2019 10:20:17",
      "content": "<p>All the kernels, which were working before is now failing with same error. Any idea?</p>",
      "votes": null,
      "replies": [
        {
          "id": 450118,
          "author_name": "s4sarath",
          "author_url": "",
          "post_date": "01/04/2019 10:32:08",
          "content": "<p>Each kernel has individual disk space right????</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 450220,
      "author_name": "s4sarath",
      "author_url": "",
      "post_date": "01/04/2019 14:13:06",
      "content": "<p>Here are some findings, if someone could help that will be great. Before going to these, i want to tell, this was working fine, but from today morning, I got into the above mentioned issues . I am attaching the screenshots</p>\n\n<h2>I.) <strong>Before model is trained</strong></h2>\n\n<p><code>1.png</code> shows the space available in <code>/kaggle/working</code></p>\n\n<p><code>2.png</code> shows the files available in <code>/kaggle/working</code></p>\n\n<p><code>3.png</code> shows the size of all files and sub directories available in <code>/kaggle/working</code> .</p>\n\n<h2>II.) <strong>After model trained for epoch 1   ( No models have been saved yet, I consciously checked it)</strong></h2>\n\n<p><code>4.png</code> shows the size of  <code>/kaggle/working</code> is <code>3.7 GB</code>. But how? I am not saving anything yet.</p>\n\n<p><code>5.png</code> shows the files available in <code>/kaggle/working</code> . No new files, same as <code>2.png</code></p>\n\n<p><code>6.png</code> is the output of <code>df -h</code>. From <code>1.png</code>, the filesystem of <code>/kaggle/working</code> is <code>/dev/loop1</code>. Now, here <code>/dev/loop1</code> is mounted on <code>/tmp</code>, and there is no <code>/kaggle/working</code>. And the size of <code>/tmp</code> is exactly same as the size of <code>/kaggle/working</code> from <code>4.png</code>. </p>\n\n<p>Finally,  <code>7.png</code> is showing the individual files and folders in <code>/kaggle</code>. Here <code>/kaggle/working</code> is present and size is just 12KB. Then how come, this disk is full error is coming.</p>\n\n<p>I know, the above is quite confusing, but this is the best way I could explain.</p>",
      "votes": null,
      "replies": [
        {
          "id": 457690,
          "author_name": "sebbovg",
          "author_url": "",
          "post_date": "01/18/2019 00:03:51",
          "content": "<p>If I follow well:</p>\n\n<p><code>4.png</code> shows <code>3.7 GB</code> used on the filesystem <code>/kaggle/working</code> is part of (not that that directory contains that usage).</p>\n\n<p>Look at what goes in <code>/tmp</code> before and after the model training.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 450239,
      "author_name": "ratthachat",
      "author_url": "",
      "post_date": "01/04/2019 14:36:34",
      "content": "<p>Cannot find any clues from : <a href=\"https://www.kaggle.com/docs/kernels\">https://www.kaggle.com/docs/kernels</a>\nOnly the last line may give some hints : A kernel may write at most 5 Gigabytes of data to disk (as output)</p>\n\n<p>Since you already remove your saved models every epoch, so I am not sure if this issue is a bug. Perhaps you should go to ask in a general forum with staffs. By the way, how big your two saved models are (before removing) ?</p>",
      "votes": null,
      "replies": [
        {
          "id": 450241,
          "author_name": "s4sarath",
          "author_url": "",
          "post_date": "01/04/2019 14:40:51",
          "content": "<p>My model size is <code>449 MB</code> . The weird thing, once I finished an epoch, before saving the model, i checked the disk space. it has consumed <code>3.7 GB / 4.9 GB</code> . But, where does that comes from? I have asked for a general support, but they will take minimum a day, so I think, my experiments are on hold. :-(</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 450247,
          "author_name": "ratthachat",
          "author_url": "",
          "post_date": "01/04/2019 14:50:10",
          "content": "<p>Okay, at least it is great that a professional will take a look at your issue. Grab some coffee or take some other experiments while waiting for some finding ;)</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 451767,
          "author_name": "ratthachat",
          "author_url": "",
          "post_date": "01/07/2019 16:23:35",
          "content": "<p>Hi Sarath, any update on this ? ;) <a href=\"/s4sarath\">@s4sarath</a></p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 452007,
          "author_name": "s4sarath",
          "author_url": "",
          "post_date": "01/08/2019 03:01:12",
          "content": "<p>Not yet. They asked me to sent the details 2 days back. I have provided this discussion link, as there is enough screenshots. Anyway, I found an alternative to save and restore, otherwise I would have been in a trouble?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 452027,
          "author_name": "ratthachat",
          "author_url": "",
          "post_date": "01/08/2019 03:57:59",
          "content": "<p>Okay, that means you can continue to experiment as you wanted. That should be enough! and it looks like your idea worked quite well too, good job!</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 452070,
          "author_name": "s4sarath",
          "author_url": "",
          "post_date": "01/08/2019 06:29:11",
          "content": "<p>Sort of, but I think it overfits though. Lets see.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "450060": "Hi Guys,\n\nI was able to commit everything perfectly. Today i am experiencing a very weird error. Committing failed with following error message. What does this mean? Can anyone be of some help?\n```\nFailure Message\nThe kernel used almost all available (5368709120) disk space. Disk writes might have failed due to insufficient space.\n```",
    "450063": "doesnt seem like a weird error to me. it seems the message is self-explanatory. isnt it? ;)",
    "450065": "Hi Abhishek, Yes it is . But 5368709120 bytes is nearly 5GB right. And I am  writing only 2 models   to disk, which I make sure, to remove after every epoch using ```shutil.rmtree```. Only the final submission file is what I am writing permanently .",
    "450110": "All the kernels, which were working before is now failing with same error. Any idea?",
    "450118": "Each kernel has individual disk space right????",
    "450220": "Here are some findings, if someone could help that will be great. Before going to these, i want to tell, this was working fine, but from today morning, I got into the above mentioned issues . I am attaching the screenshots\n\nI.) **Before model is trained**\n----------------------------------------------------------------------------------------------------------------------------\n ```1.png``` shows the space available in ```/kaggle/working```\n\n ```2.png``` shows the files available in ```/kaggle/working```\n\n ```3.png``` shows the size of all files and sub directories available in ```/kaggle/working``` .\n\n\nII.) **After model trained for epoch 1   ( No models have been saved yet, I consciously checked it)**\n----------------------------------------------------------------------------------------------------------------------------\n ```4.png``` shows the size of  ```/kaggle/working``` is ```3.7 GB```. But how? I am not saving anything yet.\n\n ```5.png``` shows the files available in ```/kaggle/working``` . No new files, same as ```2.png```\n\n ```6.png``` is the output of ```df -h```. From ```1.png```, the filesystem of ```/kaggle/working``` is ```/dev/loop1```. Now, here ```/dev/loop1``` is mounted on ```/tmp```, and there is no ```/kaggle/working```. And the size of ```/tmp``` is exactly same as the size of ```/kaggle/working``` from ```4.png```. \n\n Finally,  ```7.png``` is showing the individual files and folders in ```/kaggle```. Here ```/kaggle/working``` is present and size is just 12KB. Then how come, this disk is full error is coming.\n\nI know, the above is quite confusing, but this is the best way I could explain.",
    "450239": "Cannot find any clues from : https://www.kaggle.com/docs/kernels\nOnly the last line may give some hints : A kernel may write at most 5 Gigabytes of data to disk (as output)\n\nSince you already remove your saved models every epoch, so I am not sure if this issue is a bug. Perhaps you should go to ask in a general forum with staffs. By the way, how big your two saved models are (before removing) ?",
    "450241": "My model size is ```449 MB``` . The weird thing, once I finished an epoch, before saving the model, i checked the disk space. it has consumed ```3.7 GB / 4.9 GB``` . But, where does that comes from? I have asked for a general support, but they will take minimum a day, so I think, my experiments are on hold. :-(",
    "450247": "Okay, at least it is great that a professional will take a look at your issue. Grab some coffee or take some other experiments while waiting for some finding ;)",
    "451767": "Hi Sarath, any update on this ? ;) @s4sarath",
    "452007": "Not yet. They asked me to sent the details 2 days back. I have provided this discussion link, as there is enough screenshots. Anyway, I found an alternative to save and restore, otherwise I would have been in a trouble?",
    "452027": "Okay, that means you can continue to experiment as you wanted. That should be enough! and it looks like your idea worked quite well too, good job!",
    "452070": "Sort of, but I think it overfits though. Lets see.",
    "457690": "If I follow well:\n\n`4.png` shows `3.7 GB` used on the filesystem `/kaggle/working` is part of (not that that directory contains that usage).\n\nLook at what goes in `/tmp` before and after the model training."
  },
  "source": "meta"
}