![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2722541%2Fe38056527c130c3149ca0271ab3781a0%2Fcolab.PNG?generation=1578406691425321&alt=media\")
Kaggle GPU quota is too less to effectively train model and try out different stuff right now , and I'm already running out of my quota within 2 days. Apparently , I read online that Colab needs your PC to be on , and if it goes idle , the notebook gets shut down after 90 minutes. Is this true? If yes, how are people using Colab to train their models? I'd really appreciate if you can describe the process or if you can share a basic notebook that sets up the Kaggle API and downloading competition data :) .
", "messages": [ { "id": 711291, "postDate": "2020-01-05T22:30:42.533Z", "content": "Hello, a neat trick is to add the following snippet in the console of your browser ( I assume chrome)\nBy pressing Ctrl+Shift+I, you can open the inspector view. Then select the console tab and add the following snippets:
\n\n\nfunction ClickConnect(){\nconsole.log(\"Working\"); \ndocument.querySelector(\"colab-toolbar-button#connect\").click() \n}\nsetInterval(ClickConnect,60000)\n
This repeatedly calls the function ClickConnect (at a 60k ms interval) that will prevent it from going idle. Hope this helps!\nSource: https://medium.com/@shivamrawat_756/how-to-prevent-google-colab-from-disconnecting-717b88a128c0
", "rawMarkdown": "Hello, a neat trick is to add the following snippet in the console of your browser ( I assume chrome)\nBy pressing Ctrl+Shift+I, you can open the inspector view. Then select the console tab and add the following snippets:\n\n```\nfunction ClickConnect(){\nconsole.log(\"Working\"); \ndocument.querySelector(\"colab-toolbar-button#connect\").click() \n}\nsetInterval(ClickConnect,60000)\n```\n\nThis repeatedly calls the function ClickConnect (at a 60k ms interval) that will prevent it from going idle. Hope this helps!\nSource: https://medium.com/@shivamrawat_756/how-to-prevent-google-colab-from-disconnecting-717b88a128c0", "votes": 11, "replies": [ { "id": 711322, "postDate": "2020-01-05T23:44:41.500Z", "content": "Awesome share, thank you !!!
", "rawMarkdown": "Awesome share, thank you !!!", "votes": 1 }, { "id": 711420, "postDate": "2020-01-06T04:29:00.170Z", "content": "Thanks. This is very helpful .
", "rawMarkdown": "Thanks. This is very helpful .\n ", "votes": 1 }, { "id": 712341, "postDate": "2020-01-07T05:33:51.240Z", "content": "Bro, it is not working at least on chrome. I am still getting runtime error
", "rawMarkdown": "Bro, it is not working at least on chrome. I am still getting runtime error" }, { "id": 712539, "postDate": "2020-01-07T11:00:23.840Z", "content": "@virajbagal runtime error occurs if there is a bug in the code, not due to any network connection problem! I think you should check the traceback message.
", "rawMarkdown": "@virajbagal runtime error occurs if there is a bug in the code, not due to any network connection problem! I think you should check the traceback message.", "votes": 1 }, { "id": 712549, "postDate": "2020-01-07T11:13:51.153Z", "content": "The model runs for 2 epochs and then hangs at 3rd epoch midway and gives this error. The whole code was successfully run on Kaggle Kernel. The unusual thing is that the browser hangs and I can't traceback the error then. I tried both Chrome and Firefox.
", "rawMarkdown": "The model runs for 2 epochs and then hangs at 3rd epoch midway and gives this error. The whole code was successfully run on Kaggle Kernel. The unusual thing is that the browser hangs and I can't traceback the error then. I tried both Chrome and Firefox. ", "votes": 1 }, { "id": 712632, "postDate": "2020-01-07T13:15:04.010Z", "content": "For me his trick is working on Firefox. What is the stacktrace of the RuntimeError ?
\n\nNote that even with this trick, you are still given a maximum of 12 hours at a time on a GPU-based VM. It may be the reason why ! But 12 hours should be enough to train a model IMO. And anyway, as Deep Learning is iterative, you can always checkpoint every epochs and start from where you lastly stopped.
", "rawMarkdown": "For me his trick is working on Firefox. What is the stacktrace of the RuntimeError ? \n\nNote that even with this trick, you are still given a maximum of 12 hours at a time on a GPU-based VM. It may be the reason why ! But 12 hours should be enough to train a model IMO. And anyway, as Deep Learning is iterative, you can always checkpoint every epochs and start from where you lastly stopped. " }, { "id": 712715, "postDate": "2020-01-07T14:27:41.717Z", "content": "\n\nAfter this the browser hangs, so I cannot trace back the error further. Please help. This happens almost immediately after 30 mins ,i.e, midway of 3rd epoch. The whole code runs well on Kaggle Kernel, so there is no error in the code
", "rawMarkdown": "@udaykamal \n\n\nAfter this the browser hangs, so I cannot trace back the error further. Please help. This happens almost immediately after 30 mins ,i.e, midway of 3rd epoch. The whole code runs well on Kaggle Kernel, so there is no error in the code", "votes": 1 }, { "id": 713483, "postDate": "2020-01-08T10:33:55.950Z", "content": "@virajbagal I am not quite sure about the reason behind this error. A quick google search led me to this github issue which you can look into:\nhttps://github.com/googlecolab/colabtools/issues/584\nHowever, it seems that you have already found a workaround, so all the best!
", "rawMarkdown": "@virajbagal I am not quite sure about the reason behind this error. A quick google search led me to this github issue which you can look into:\nhttps://github.com/googlecolab/colabtools/issues/584\nHowever, it seems that you have already found a workaround, so all the best! ", "votes": 1 }, { "id": 961281, "postDate": "2020-08-07T04:37:17.113Z", "content": "this was so much useful. thank you @udaykamal . It ran completely fine on chrome, but I had to put these lines of code in console every 3-4 epochs (1-2 hr) to minimise the casualties of getting stuck in runtime error at the edge of the last epoch.
", "rawMarkdown": "this was so much useful. thank you @udaykamal . It ran completely fine on chrome, but I had to put these lines of code in console every 3-4 epochs (1-2 hr) to minimise the casualties of getting stuck in runtime error at the edge of the last epoch." } ] }, { "id": 711074, "postDate": "2020-01-05T16:51:17.820Z", "content": "You can also try this:
\n\n\n \n", "rawMarkdown": "You can also try this:\n> https://gradient.paperspace.com/free-gpu\n\n", "votes": 5, "replies": [ { "id": 711078, "postDate": "2020-01-05T16:57:50.207Z", "content": "
Will check it out , thanks :)
", "rawMarkdown": "Will check it out , thanks :)" } ] }, { "id": 712779, "postDate": "2020-01-07T15:15:29.950Z", "content": "@p4rallax Colab Auto Reconnect Google Chrome extension. It's working like a charm for me.
", "rawMarkdown": "@p4rallax Colab Auto Reconnect Google Chrome extension. It's working like a charm for me.", "votes": 6, "replies": [ { "id": 712822, "postDate": "2020-01-07T16:02:08.597Z", "content": "Sounds convenient. Thanks :)
", "rawMarkdown": "Sounds convenient. Thanks :)" }, { "id": 713470, "postDate": "2020-01-08T10:07:36.007Z", "content": "Finally, this worked for me. Thanks
", "rawMarkdown": "Finally, this worked for me. Thanks", "votes": 1 } ] }, { "id": 711688, "postDate": "2020-01-06T12:04:32.193Z", "content": "Here is a code snippet for downloading competition data on colab and extract them in a directory (with fuse-zip).
\n\nThe first 2 lines (where i uninstall and reinstall kaggle) are because kaggle api are not up to data on google colab notebooks.
\n\n```\n!pip uninstall -y kaggle\n!pip install kaggle\n!apt-get install fuse-zip
\n\nimport os
\n\nos.environ[\"KAGGLE_USERNAME\"] = \"your_username\"\nos.environ[\"KAGGLE_KEY\"] = \"your_token\"
\n\nraw_data_dir = \"input/raw\"\n!rm -r {raw_data_dir}\n!kaggle competitions download -c bengaliai-cv19 -p {raw_data_dir}
\n\n!mkdir ../input\n!mkdir {input_dir}
\n\n!fuse-zip {raw_data_dir}/bengaliai-cv19.zip {input_dir}\n```
\n\nYou can get your KAGGLE_USERNAME/_KEY from your kaggle profile.
", "rawMarkdown": "Here is a code snippet for downloading competition data on colab and extract them in a directory (with fuse-zip).\n\nThe first 2 lines (where i uninstall and reinstall kaggle) are because kaggle api are not up to data on google colab notebooks.\n\n```\n!pip uninstall -y kaggle\n!pip install kaggle\n!apt-get install fuse-zip\n\nimport os\n\nos.environ[\"KAGGLE_USERNAME\"] = \"your_username\"\nos.environ[\"KAGGLE_KEY\"] = \"your_token\"\n\nraw_data_dir = \"input/raw\"\n!rm -r {raw_data_dir}\n!kaggle competitions download -c bengaliai-cv19 -p {raw_data_dir}\n\n!mkdir ../input\n!mkdir {input_dir}\n\n!fuse-zip {raw_data_dir}/bengaliai-cv19.zip {input_dir}\n```\n\nYou can get your KAGGLE_USERNAME/_KEY from your kaggle profile.", "votes": 6 }, { "id": 711065, "postDate": "2020-01-05T16:36:25.763Z", "content": "Kaggle GPU quota is too less to effectively train model and try out different stuff right now , and I'm already running out of my quota within 2 days. Apparently , I read online that Colab needs your PC to be on , and if it goes idle , the notebook gets shut down after 90 minutes. Is this true? If yes, how are people using Colab to train their models? I'd really appreciate if you can describe the process or if you can share a basic notebook that sets up the Kaggle API and downloading competition data :) .
", "rawMarkdown": "Kaggle GPU quota is too less to effectively train model and try out different stuff right now , and I'm already running out of my quota within 2 days. Apparently , I read online that Colab needs your PC to be on , and if it goes idle , the notebook gets shut down after 90 minutes. Is this true? If yes, how are people using Colab to train their models? I'd really appreciate if you can describe the process or if you can share a basic notebook that sets up the Kaggle API and downloading competition data :) .", "votes": 5 }, { "id": 713220, "postDate": "2020-01-08T03:14:54.273Z", "content": "Yes that's true that colab server gets shut down after 30min idle session.\n1. You can try this > Method to prevent getting shut down colab notebook.\n2. See this > @saedhussain/google-colaboratory-and-kaggle-datasets-b57a83eb6ef8\">Article for Kaggle API and downloading competition data
", "rawMarkdown": "Yes that's true that colab server gets shut down after 30min idle session.\n1. You can try this > [Method](https://medium.com/@shivamrawat_756/how-to-prevent-google-colab-from-disconnecting-717b88a128c0) to prevent getting shut down colab notebook.\n2. See this > [Article](https://medium.com/@saedhussain/google-colaboratory-and-kaggle-datasets-b57a83eb6ef8) for Kaggle API and downloading competition data", "votes": 3 }, { "id": 742188, "postDate": "2020-02-11T06:17:26.323Z", "content": "This discussion helped a lot @p4rallax
", "rawMarkdown": "This discussion helped a lot @p4rallax ", "votes": 1 }, { "id": 719492, "postDate": "2020-01-15T14:50:21.667Z", "content": "Quick question: By using the Colab Auto Reconnect Google Chrome extension, if my notebook runs for more than 12 hours (and gets interrupted at 12th hour) will I get a ban or something? If not, I can just save my models and rerun the notebook again and again.\nWhy is google allowing this? Where is the catch?
", "rawMarkdown": "Quick question: By using the Colab Auto Reconnect Google Chrome extension, if my notebook runs for more than 12 hours (and gets interrupted at 12th hour) will I get a ban or something? If not, I can just save my models and rerun the notebook again and again.\nWhy is google allowing this? Where is the catch? ", "votes": 1, "replies": [ { "id": 719497, "postDate": "2020-01-15T14:55:31.073Z", "content": "You will not allowed to connect to a runtime with a GPU backend for 8 hours. Anyway you can download the weights and continue training with another Google account.
", "rawMarkdown": "You will not allowed to connect to a runtime with a GPU backend for 8 hours. Anyway you can download the weights and continue training with another Google account. ", "votes": 2 }, { "id": 719622, "postDate": "2020-01-15T16:54:20.600Z", "content": "@timetraveller98 wanna team up?
", "rawMarkdown": "@timetraveller98 wanna team up?" }, { "id": 719651, "postDate": "2020-01-15T17:31:45.367Z", "content": "@p4rallax Thank you for asking but I think its too soon for me to team up right now. Also, I haven't made a single submission yet so I am not sure how good I may perform. (Local CV isn't going very good for me right now either)
", "rawMarkdown": "@p4rallax Thank you for asking but I think its too soon for me to team up right now. Also, I haven't made a single submission yet so I am not sure how good I may perform. (Local CV isn't going very good for me right now either)", "votes": 1 }, { "id": 719732, "postDate": "2020-01-15T19:42:04.677Z", "content": "Sure , np. Good luck! :D
", "rawMarkdown": "Sure , np. Good luck! :D", "votes": 1 } ] }, { "id": 713175, "postDate": "2020-01-08T01:56:16.413Z", "content": "My experience with colab is that I use web browser of my cell phone to open colab and keep it running using cell phone. With that way, you save energy on your PC and you can have a look in the trainning process anytime and anywhere using your cellphone.
", "rawMarkdown": "My experience with colab is that I use web browser of my cell phone to open colab and keep it running using cell phone. With that way, you save energy on your PC and you can have a look in the trainning process anytime and anywhere using your cellphone.", "votes": 1 }, { "id": 732124, "postDate": "2020-01-29T14:10:24.260Z", "content": "I try to use this data on Colab, but get this problem:OSError: [Errno 5] Input/output error:\nHow to solve this problem？
", "rawMarkdown": "I try to use [this data](https://www.kaggle.com/pestipeti/bengaliai) on Colab, but get this problem:*OSError: [Errno 5] Input/output error:*\nHow to solve this problem？", "replies": [ { "id": 732132, "postDate": "2020-01-29T14:23:00.893Z", "content": "That's the issue , that is around 200k files you need to extract into your Drive to use in Colab and it crashes since it cannot handle so many files. We had the same problem and gave up on it. Either make it into a .feather format data and just 4 files like the original data , or just train on Kaggle. Those seem to be the only possible solutions :/
", "rawMarkdown": "That's the issue , that is around 200k files you need to extract into your Drive to use in Colab and it crashes since it cannot handle so many files. We had the same problem and gave up on it. Either make it into a .feather format data and just 4 files like the original data , or just train on Kaggle. Those seem to be the only possible solutions :/" }, { "id": 732226, "postDate": "2020-01-29T15:33:51.360Z", "content": "You can try the method posted by @dimartinot. He use cv.imdecode on the zipfile instead of cv.imread on upzipfile. But it seems slower😷 .
", "rawMarkdown": "You can try [the method ](https://www.kaggle.com/c/bengaliai-cv19/discussion/126025#720711)posted by @dimartinot. He use *cv.imdecode* on the zipfile instead of *cv.imread* on upzipfile. But it seems slower😷 .", "votes": 1 }, { "id": 732317, "postDate": "2020-01-29T17:16:27.827Z", "content": "\nimport os\n!kaggle datasets download iafoss/grapheme-imgs-128x128\nif not os.path.isfile('train/Train_0.png'):\n !unzip -q grapheme-imgs-128x128 -d train/\n\nThis works for me using this dataset. Perhaps its the amount of folders in that dataset?
I used a dataset I created , it had no folders and the zip file only had the 200k images. I'm extracting the files in a similar way , I'll try this out and let you know :)
", "rawMarkdown": "I used a dataset I created , it had no folders and the zip file only had the 200k images. I'm extracting the files in a similar way , I'll try this out and let you know :)" } ] }, { "id": 716198, "postDate": "2020-01-11T11:52:47.533Z", "content": "Is my local GPU-nvidia 940 MX sufficent for this competition?
", "rawMarkdown": "Is my local GPU-nvidia 940 MX sufficent for this competition?", "replies": [ { "id": 732309, "postDate": "2020-01-29T17:04:21.033Z", "content": "940MX is a very weak graphics card, it is definitely not recommended to use it for neural networks. I struggle even to run simple networks on this graphics card. You would definitely be better off using colab.
", "rawMarkdown": "940MX is a very weak graphics card, it is definitely not recommended to use it for neural networks. I struggle even to run simple networks on this graphics card. You would definitely be better off using colab. " } ] }, { "id": 713215, "postDate": "2020-01-08T03:05:12.273Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 711291, "author_name": "Uday Kamal", "author_url": "", "post_date": "2020-01-05T22:30:42.533000", "content": "Hello, a neat trick is to add the following snippet in the console of your browser ( I assume chrome)\nBy pressing Ctrl+Shift+I, you can open the inspector view. Then select the console tab and add the following snippets:
\n\n\nfunction ClickConnect(){\nconsole.log(\"Working\"); \ndocument.querySelector(\"colab-toolbar-button#connect\").click() \n}\nsetInterval(ClickConnect,60000)\n
This repeatedly calls the function ClickConnect (at a 60k ms interval) that will prevent it from going idle. Hope this helps!\nSource: https://medium.com/@shivamrawat_756/how-to-prevent-google-colab-from-disconnecting-717b88a128c0
", "votes": 11, "replies": [ { "id": 711322, "author_name": "Thomas Di Martino", "author_url": "", "post_date": "2020-01-05T23:44:41.500000", "content": "Awesome share, thank you !!!
", "votes": 1, "replies": [] }, { "id": 711420, "author_name": "Viraj Bagal", "author_url": "", "post_date": "2020-01-06T04:29:00.170000", "content": "Thanks. This is very helpful .
", "votes": 1, "replies": [] }, { "id": 712341, "author_name": "Viraj Bagal", "author_url": "", "post_date": "2020-01-07T05:33:51.240000", "content": "Bro, it is not working at least on chrome. I am still getting runtime error
", "votes": 0, "replies": [] }, { "id": 712539, "author_name": "Uday Kamal", "author_url": "", "post_date": "2020-01-07T11:00:23.840000", "content": "@virajbagal runtime error occurs if there is a bug in the code, not due to any network connection problem! I think you should check the traceback message.
", "votes": 1, "replies": [] }, { "id": 712549, "author_name": "Viraj Bagal", "author_url": "", "post_date": "2020-01-07T11:13:51.153000", "content": "The model runs for 2 epochs and then hangs at 3rd epoch midway and gives this error. The whole code was successfully run on Kaggle Kernel. The unusual thing is that the browser hangs and I can't traceback the error then. I tried both Chrome and Firefox.
", "votes": 1, "replies": [] }, { "id": 712632, "author_name": "Thomas Di Martino", "author_url": "", "post_date": "2020-01-07T13:15:04.010000", "content": "For me his trick is working on Firefox. What is the stacktrace of the RuntimeError ?
\n\nNote that even with this trick, you are still given a maximum of 12 hours at a time on a GPU-based VM. It may be the reason why ! But 12 hours should be enough to train a model IMO. And anyway, as Deep Learning is iterative, you can always checkpoint every epochs and start from where you lastly stopped.
", "votes": 0, "replies": [] }, { "id": 712715, "author_name": "Viraj Bagal", "author_url": "", "post_date": "2020-01-07T14:27:41.717000", "content": "\n\nAfter this the browser hangs, so I cannot trace back the error further. Please help. This happens almost immediately after 30 mins ,i.e, midway of 3rd epoch. The whole code runs well on Kaggle Kernel, so there is no error in the code
", "votes": 1, "replies": [] }, { "id": 713483, "author_name": "Uday Kamal", "author_url": "", "post_date": "2020-01-08T10:33:55.950000", "content": "@virajbagal I am not quite sure about the reason behind this error. A quick google search led me to this github issue which you can look into:\nhttps://github.com/googlecolab/colabtools/issues/584\nHowever, it seems that you have already found a workaround, so all the best!
", "votes": 1, "replies": [] }, { "id": 961281, "author_name": "Shreya Jain", "author_url": "", "post_date": "2020-08-07T04:37:17.113000", "content": "this was so much useful. thank you @udaykamal . It ran completely fine on chrome, but I had to put these lines of code in console every 3-4 epochs (1-2 hr) to minimise the casualties of getting stuck in runtime error at the edge of the last epoch.
", "votes": 0, "replies": [] } ] }, { "id": 711074, "author_name": "Bibek", "author_url": "", "post_date": "2020-01-05T16:51:17.820000", "content": "You can also try this:
\n\n\n \n", "votes": 5, "replies": [ { "id": 711078, "author_name": "Satwik", "author_url": "", "post_date": "2020-01-05T16:57:50.207000", "content": "
Will check it out , thanks :)
", "votes": 0, "replies": [] } ] }, { "id": 712779, "author_name": "Mourad", "author_url": "", "post_date": "2020-01-07T15:15:29.950000", "content": "@p4rallax Colab Auto Reconnect Google Chrome extension. It's working like a charm for me.
", "votes": 6, "replies": [ { "id": 712822, "author_name": "Satwik", "author_url": "", "post_date": "2020-01-07T16:02:08.597000", "content": "Sounds convenient. Thanks :)
", "votes": 0, "replies": [] }, { "id": 713470, "author_name": "Viraj Bagal", "author_url": "", "post_date": "2020-01-08T10:07:36.007000", "content": "Finally, this worked for me. Thanks
", "votes": 1, "replies": [] } ] }, { "id": 711688, "author_name": "LAZCoder", "author_url": "", "post_date": "2020-01-06T12:04:32.193000", "content": "Here is a code snippet for downloading competition data on colab and extract them in a directory (with fuse-zip).
\n\nThe first 2 lines (where i uninstall and reinstall kaggle) are because kaggle api are not up to data on google colab notebooks.
\n\n```\n!pip uninstall -y kaggle\n!pip install kaggle\n!apt-get install fuse-zip
\n\nimport os
\n\nos.environ[\"KAGGLE_USERNAME\"] = \"your_username\"\nos.environ[\"KAGGLE_KEY\"] = \"your_token\"
\n\nraw_data_dir = \"input/raw\"\n!rm -r {raw_data_dir}\n!kaggle competitions download -c bengaliai-cv19 -p {raw_data_dir}
\n\n!mkdir ../input\n!mkdir {input_dir}
\n\n!fuse-zip {raw_data_dir}/bengaliai-cv19.zip {input_dir}\n```
\n\nYou can get your KAGGLE_USERNAME/_KEY from your kaggle profile.
", "votes": 6, "replies": [] }, { "id": 713220, "author_name": "SunWalker", "author_url": "", "post_date": "2020-01-08T03:14:54.273000", "content": "Yes that's true that colab server gets shut down after 30min idle session.\n1. You can try this > Method to prevent getting shut down colab notebook.\n2. See this > @saedhussain/google-colaboratory-and-kaggle-datasets-b57a83eb6ef8\">Article for Kaggle API and downloading competition data
", "votes": 3, "replies": [] }, { "id": 742188, "author_name": "Viraj Bagal", "author_url": "", "post_date": "2020-02-11T06:17:26.323000", "content": "This discussion helped a lot @p4rallax
", "votes": 1, "replies": [] }, { "id": 719492, "author_name": "timetraveller", "author_url": "", "post_date": "2020-01-15T14:50:21.667000", "content": "Quick question: By using the Colab Auto Reconnect Google Chrome extension, if my notebook runs for more than 12 hours (and gets interrupted at 12th hour) will I get a ban or something? If not, I can just save my models and rerun the notebook again and again.\nWhy is google allowing this? Where is the catch?
", "votes": 1, "replies": [ { "id": 719497, "author_name": "LAZCoder", "author_url": "", "post_date": "2020-01-15T14:55:31.073000", "content": "You will not allowed to connect to a runtime with a GPU backend for 8 hours. Anyway you can download the weights and continue training with another Google account.
", "votes": 2, "replies": [] }, { "id": 719622, "author_name": "Satwik", "author_url": "", "post_date": "2020-01-15T16:54:20.600000", "content": "@timetraveller98 wanna team up?
", "votes": 0, "replies": [] }, { "id": 719651, "author_name": "timetraveller", "author_url": "", "post_date": "2020-01-15T17:31:45.367000", "content": "@p4rallax Thank you for asking but I think its too soon for me to team up right now. Also, I haven't made a single submission yet so I am not sure how good I may perform. (Local CV isn't going very good for me right now either)
", "votes": 1, "replies": [] }, { "id": 719732, "author_name": "Satwik", "author_url": "", "post_date": "2020-01-15T19:42:04.677000", "content": "Sure , np. Good luck! :D
", "votes": 1, "replies": [] } ] }, { "id": 713175, "author_name": "Truong Tuan Minh", "author_url": "", "post_date": "2020-01-08T01:56:16.413000", "content": "My experience with colab is that I use web browser of my cell phone to open colab and keep it running using cell phone. With that way, you save energy on your PC and you can have a look in the trainning process anytime and anywhere using your cellphone.
", "votes": 1, "replies": [] }, { "id": 732124, "author_name": "ld", "author_url": "", "post_date": "2020-01-29T14:10:24.260000", "content": "I try to use this data on Colab, but get this problem:OSError: [Errno 5] Input/output error:\nHow to solve this problem？
", "votes": 0, "replies": [ { "id": 732132, "author_name": "Satwik", "author_url": "", "post_date": "2020-01-29T14:23:00.893000", "content": "That's the issue , that is around 200k files you need to extract into your Drive to use in Colab and it crashes since it cannot handle so many files. We had the same problem and gave up on it. Either make it into a .feather format data and just 4 files like the original data , or just train on Kaggle. Those seem to be the only possible solutions :/
", "votes": 0, "replies": [] }, { "id": 732226, "author_name": "ld", "author_url": "", "post_date": "2020-01-29T15:33:51.360000", "content": "You can try the method posted by @dimartinot. He use cv.imdecode on the zipfile instead of cv.imread on upzipfile. But it seems slower😷 .
", "votes": 1, "replies": [] }, { "id": 732317, "author_name": "GreatGameDota", "author_url": "", "post_date": "2020-01-29T17:16:27.827000", "content": "\nimport os\n!kaggle datasets download iafoss/grapheme-imgs-128x128\nif not os.path.isfile('train/Train_0.png'):\n !unzip -q grapheme-imgs-128x128 -d train/\n\nThis works for me using this dataset. Perhaps its the amount of folders in that dataset?
I used a dataset I created , it had no folders and the zip file only had the 200k images. I'm extracting the files in a similar way , I'll try this out and let you know :)
", "votes": 0, "replies": [] } ] }, { "id": 716198, "author_name": "Kurian Benoy", "author_url": "", "post_date": "2020-01-11T11:52:47.533000", "content": "Is my local GPU-nvidia 940 MX sufficent for this competition?
", "votes": 0, "replies": [ { "id": 732309, "author_name": "G Sriram", "author_url": "", "post_date": "2020-01-29T17:04:21.033000", "content": "940MX is a very weak graphics card, it is definitely not recommended to use it for neural networks. I struggle even to run simple networks on this graphics card. You would definitely be better off using colab.
", "votes": 0, "replies": [] } ] }, { "id": 713215, "author_name": "", "author_url": "", "post_date": "2020-01-08T03:05:12.273000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "711291": "Hello, a neat trick is to add the following snippet in the console of your browser ( I assume chrome)\nBy pressing Ctrl+Shift+I, you can open the inspector view. Then select the console tab and add the following snippets:\n\n```\nfunction ClickConnect(){\nconsole.log(\"Working\"); \ndocument.querySelector(\"colab-toolbar-button#connect\").click() \n}\nsetInterval(ClickConnect,60000)\n```\n\nThis repeatedly calls the function ClickConnect (at a 60k ms interval) that will prevent it from going idle. Hope this helps!\nSource: https://medium.com/@shivamrawat_756/how-to-prevent-google-colab-from-disconnecting-717b88a128c0", "711074": "You can also try this:\n> https://gradient.paperspace.com/free-gpu\n\n", "712779": "@p4rallax Colab Auto Reconnect Google Chrome extension. It's working like a charm for me.", "711688": "Here is a code snippet for downloading competition data on colab and extract them in a directory (with fuse-zip).\n\nThe first 2 lines (where i uninstall and reinstall kaggle) are because kaggle api are not up to data on google colab notebooks.\n\n```\n!pip uninstall -y kaggle\n!pip install kaggle\n!apt-get install fuse-zip\n\nimport os\n\nos.environ[\"KAGGLE_USERNAME\"] = \"your_username\"\nos.environ[\"KAGGLE_KEY\"] = \"your_token\"\n\nraw_data_dir = \"input/raw\"\n!rm -r {raw_data_dir}\n!kaggle competitions download -c bengaliai-cv19 -p {raw_data_dir}\n\n!mkdir ../input\n!mkdir {input_dir}\n\n!fuse-zip {raw_data_dir}/bengaliai-cv19.zip {input_dir}\n```\n\nYou can get your KAGGLE_USERNAME/_KEY from your kaggle profile.", "711065": "Kaggle GPU quota is too less to effectively train model and try out different stuff right now , and I'm already running out of my quota within 2 days. Apparently , I read online that Colab needs your PC to be on , and if it goes idle , the notebook gets shut down after 90 minutes. Is this true? If yes, how are people using Colab to train their models? I'd really appreciate if you can describe the process or if you can share a basic notebook that sets up the Kaggle API and downloading competition data :) .", "713220": "Yes that's true that colab server gets shut down after 30min idle session.\n1. You can try this > [Method](https://medium.com/@shivamrawat_756/how-to-prevent-google-colab-from-disconnecting-717b88a128c0) to prevent getting shut down colab notebook.\n2. See this > [Article](https://medium.com/@saedhussain/google-colaboratory-and-kaggle-datasets-b57a83eb6ef8) for Kaggle API and downloading competition data", "742188": "This discussion helped a lot @p4rallax ", "719492": "Quick question: By using the Colab Auto Reconnect Google Chrome extension, if my notebook runs for more than 12 hours (and gets interrupted at 12th hour) will I get a ban or something? If not, I can just save my models and rerun the notebook again and again.\nWhy is google allowing this? Where is the catch? ", "713175": "My experience with colab is that I use web browser of my cell phone to open colab and keep it running using cell phone. With that way, you save energy on your PC and you can have a look in the trainning process anytime and anywhere using your cellphone.", "732124": "I try to use [this data](https://www.kaggle.com/pestipeti/bengaliai) on Colab, but get this problem:*OSError: [Errno 5] Input/output error:*\nHow to solve this problem？", "716198": "Is my local GPU-nvidia 940 MX sufficent for this competition?", "713215": "" } }