Hi!
\n\nSo, like a lot of us I'm trying to get the most out of this competition before the end of it. Kaggle notebooks are the main resource I use for my coding, as I don't have access to a good enough GPU to get reasonable compute times.\nLately, I've been working on a classifier to try to improve my overall score, but Kaggle notebooks keep shutting down for some reason. It's probably not a memory error, I've managed to get it running a few times all the way without it crashing.
\n\nThe weirdest part of all of this is that these notebooks still seem to be running. I have setup a function that sends me logs on Slack, so I can see my training results while not having to stay glued to my laptop no stop. And even after the notebook shuts down, I sometimes still get these messages.
\n\nThis is really frustrating, as I'm basically dumping precious GPU hours through the window.\nHas anybody else experienced this? And if so, would anybody have an idea on how to fix this?
\n\nThanks!
", "messages": [ { "id": "672921", "postDate": "11/14/2019 09:36:06", "content": "Hi!
\n\nSo, like a lot of us I'm trying to get the most out of this competition before the end of it. Kaggle notebooks are the main resource I use for my coding, as I don't have access to a good enough GPU to get reasonable compute times.\nLately, I've been working on a classifier to try to improve my overall score, but Kaggle notebooks keep shutting down for some reason. It's probably not a memory error, I've managed to get it running a few times all the way without it crashing.
\n\nThe weirdest part of all of this is that these notebooks still seem to be running. I have setup a function that sends me logs on Slack, so I can see my training results while not having to stay glued to my laptop no stop. And even after the notebook shuts down, I sometimes still get these messages.
\n\nThis is really frustrating, as I'm basically dumping precious GPU hours through the window.\nHas anybody else experienced this? And if so, would anybody have an idea on how to fix this?
\n\nThanks!
", "rawMarkdown": "Hi!\n\nSo, like a lot of us I'm trying to get the most out of this competition before the end of it. Kaggle notebooks are the main resource I use for my coding, as I don't have access to a good enough GPU to get reasonable compute times.\nLately, I've been working on a classifier to try to improve my overall score, but Kaggle notebooks keep shutting down for some reason. It's probably not a memory error, I've managed to get it running a few times all the way without it crashing.\n\nThe weirdest part of all of this is that these notebooks still seem to be running. I have setup a function that sends me logs on Slack, so I can see my training results while not having to stay glued to my laptop no stop. And even after the notebook shuts down, I sometimes still get these messages.\n\nThis is really frustrating, as I'm basically dumping precious GPU hours through the window.\nHas anybody else experienced this? And if so, would anybody have an idea on how to fix this?\n\nThanks!", "votes": null }, { "id": "673487", "postDate": "11/15/2019 03:40:26", "content": "Try running then committing
", "rawMarkdown": "Try running then committing", "votes": null }, { "id": "673500", "postDate": "11/15/2019 04:20:02", "content": "Yes. I have noticed that interactive Kaggle notebooks sometimes unexpectedly turn off. It's frustrating because you can lose your work. As an alternative, you can consider using Google Colab's free GPUs. I posted a \"how to\" here
", "rawMarkdown": "Yes. I have noticed that interactive Kaggle notebooks sometimes unexpectedly turn off. It's frustrating because you can lose your work. As an alternative, you can consider using Google Colab's free GPUs. I posted a \"how to\" [here][1]\n\n[1]: https://www.kaggle.com/c/understanding_cloud_organization/discussion/117021", "votes": null }, { "id": "673587", "postDate": "11/15/2019 07:36:20", "content": "Thanks for the reply!\nI've also been using Colab, but I would like to run Kaggle notebooks and Colab, to try multiple things at the same time. And Colab also has problems of its own.\nI'm also using Paperspace as they have a free GPU tier, for those that might be interested, it's an interesting option too!
", "rawMarkdown": "Thanks for the reply!\nI've also been using Colab, but I would like to run Kaggle notebooks *and* Colab, to try multiple things at the same time. And Colab also has problems of its own.\nI'm also using Paperspace as they have a free GPU tier, for those that might be interested, it's an interesting option too!", "votes": null }, { "id": "673588", "postDate": "11/15/2019 07:36:51", "content": "That could be one solution, but it doesn't really allow for tinkering then, which is still what I'd like to be doing.
", "rawMarkdown": "That could be one solution, but it doesn't really allow for tinkering then, which is still what I'd like to be doing.", "votes": null }, { "id": "673682", "postDate": "11/15/2019 10:30:31", "content": "Try to decrease memory usage. During RSNA competition I decreased the batch size and it helped.
", "rawMarkdown": "Try to decrease memory usage. During RSNA competition I decreased the batch size and it helped.", "votes": null }, { "id": "673728", "postDate": "11/15/2019 12:04:12", "content": "I have met this problem several times, maybe it's the network. I am in china, after switched vpn, things get normal.\nIf you can't solve this, I think the best way is just \"commit\", you can change your code to save time, like rearranging code order, that's what I did. Good luck!
", "rawMarkdown": "I have met this problem several times, maybe it's the network. I am in china, after switched vpn, things get normal.\nIf you can't solve this, I think the best way is just \"commit\", you can change your code to save time, like rearranging code order, that's what I did. Good luck!", "votes": null }, { "id": "673729", "postDate": "11/15/2019 12:05:16", "content": "It doesn't seem to have to do anything to do with this.\nOnce again, running the same code sometimes works, other times it just shuts down. The kernel doesn't crash, it shuts down.\nHence my post, asking if people know why these happen?
", "rawMarkdown": "It doesn't seem to have to do anything to do with this.\nOnce again, running the same code sometimes works, other times it just shuts down. The kernel doesn't crash, it shuts down.\nHence my post, asking if people know why these happen?", "votes": null }, { "id": "673730", "postDate": "11/15/2019 12:06:13", "content": "Oh, interesting\nI'm in the Netherlands, with a wired connection, so really good networking.\nBut thanks for the return of experience!
", "rawMarkdown": "Oh, interesting\nI'm in the Netherlands, with a wired connection, so really good networking.\nBut thanks for the return of experience!", "votes": null } ], "comments": [ { "id": 673487, "author_name": "krishnakatyal", "author_url": "", "post_date": "11/15/2019 03:40:26", "content": "Try running then committing
", "votes": null, "replies": [ { "id": 673588, "author_name": "maxlenormand", "author_url": "", "post_date": "11/15/2019 07:36:51", "content": "That could be one solution, but it doesn't really allow for tinkering then, which is still what I'd like to be doing.
", "votes": null, "replies": [] } ] }, { "id": 673500, "author_name": "cdeotte", "author_url": "", "post_date": "11/15/2019 04:20:02", "content": "Yes. I have noticed that interactive Kaggle notebooks sometimes unexpectedly turn off. It's frustrating because you can lose your work. As an alternative, you can consider using Google Colab's free GPUs. I posted a \"how to\" here
", "votes": null, "replies": [ { "id": 673587, "author_name": "maxlenormand", "author_url": "", "post_date": "11/15/2019 07:36:20", "content": "Thanks for the reply!\nI've also been using Colab, but I would like to run Kaggle notebooks and Colab, to try multiple things at the same time. And Colab also has problems of its own.\nI'm also using Paperspace as they have a free GPU tier, for those that might be interested, it's an interesting option too!
", "votes": null, "replies": [] } ] }, { "id": 673682, "author_name": "tieros", "author_url": "", "post_date": "11/15/2019 10:30:31", "content": "Try to decrease memory usage. During RSNA competition I decreased the batch size and it helped.
", "votes": null, "replies": [ { "id": 673729, "author_name": "maxlenormand", "author_url": "", "post_date": "11/15/2019 12:05:16", "content": "It doesn't seem to have to do anything to do with this.\nOnce again, running the same code sometimes works, other times it just shuts down. The kernel doesn't crash, it shuts down.\nHence my post, asking if people know why these happen?
", "votes": null, "replies": [] } ] }, { "id": 673728, "author_name": "zhaoguowang666", "author_url": "", "post_date": "11/15/2019 12:04:12", "content": "I have met this problem several times, maybe it's the network. I am in china, after switched vpn, things get normal.\nIf you can't solve this, I think the best way is just \"commit\", you can change your code to save time, like rearranging code order, that's what I did. Good luck!
", "votes": null, "replies": [ { "id": 673730, "author_name": "maxlenormand", "author_url": "", "post_date": "11/15/2019 12:06:13", "content": "Oh, interesting\nI'm in the Netherlands, with a wired connection, so really good networking.\nBut thanks for the return of experience!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "672921": "Hi!\n\nSo, like a lot of us I'm trying to get the most out of this competition before the end of it. Kaggle notebooks are the main resource I use for my coding, as I don't have access to a good enough GPU to get reasonable compute times.\nLately, I've been working on a classifier to try to improve my overall score, but Kaggle notebooks keep shutting down for some reason. It's probably not a memory error, I've managed to get it running a few times all the way without it crashing.\n\nThe weirdest part of all of this is that these notebooks still seem to be running. I have setup a function that sends me logs on Slack, so I can see my training results while not having to stay glued to my laptop no stop. And even after the notebook shuts down, I sometimes still get these messages.\n\nThis is really frustrating, as I'm basically dumping precious GPU hours through the window.\nHas anybody else experienced this? And if so, would anybody have an idea on how to fix this?\n\nThanks!", "673487": "Try running then committing", "673500": "Yes. I have noticed that interactive Kaggle notebooks sometimes unexpectedly turn off. It's frustrating because you can lose your work. As an alternative, you can consider using Google Colab's free GPUs. I posted a \"how to\" [here][1]\n\n[1]: https://www.kaggle.com/c/understanding_cloud_organization/discussion/117021", "673587": "Thanks for the reply!\nI've also been using Colab, but I would like to run Kaggle notebooks *and* Colab, to try multiple things at the same time. And Colab also has problems of its own.\nI'm also using Paperspace as they have a free GPU tier, for those that might be interested, it's an interesting option too!", "673588": "That could be one solution, but it doesn't really allow for tinkering then, which is still what I'd like to be doing.", "673682": "Try to decrease memory usage. During RSNA competition I decreased the batch size and it helped.", "673728": "I have met this problem several times, maybe it's the network. I am in china, after switched vpn, things get normal.\nIf you can't solve this, I think the best way is just \"commit\", you can change your code to save time, like rearranging code order, that's what I did. Good luck!", "673729": "It doesn't seem to have to do anything to do with this.\nOnce again, running the same code sometimes works, other times it just shuts down. The kernel doesn't crash, it shuts down.\nHence my post, asking if people know why these happen?", "673730": "Oh, interesting\nI'm in the Netherlands, with a wired connection, so really good networking.\nBut thanks for the return of experience!" }, "source": "meta" }