{ "id": 126025, "title": "Problems with Colab", "url": "/competitions/bengaliai-cv19/discussion/126025", "author_name": "", "post_date": "2020-01-15T08:12:50.302912Z", "votes": 2, "comment_count": 19, "views": 0, "content": "", "messages": [ { "id": "719186", "postDate": "01/15/2020 08:12:50", "content": "", "rawMarkdown": "Because limited quota of kaggle, I move to use Colab for training my model. Everything seems ok when I trained small data, but when I train all of the data, the Colab have been disconnected after 1 epoch and trying reconnect caused my Chrome crash (my ram increased very fast and I did not know why). Did anybody have the same issue and How could I do to solve this problems. Thank you all!", "votes": null }, { "id": "719240", "postDate": "01/15/2020 09:30:00", "content": "

Make sure you use GPU enabled runtime. Apart from that try with\n- smaller batches\n- smaller image size\n- simpler network\nand increase progressively to see how it performs.

\n\n

From my experience a CNN net with about Trainable params: 18,000,000, image_size=128x128 and batch_size=64 runs in about 20 mins per epoch.

", "rawMarkdown": "Make sure you use GPU enabled runtime. Apart from that try with\n- smaller batches\n- smaller image size\n- simpler network\nand increase progressively to see how it performs. \n\nFrom my experience a CNN net with about `Trainable params: 18,000,000`, `image_size=128x128` and `batch_size=64` runs in about 20 mins per epoch.", "votes": null }, { "id": "719243", "postDate": "01/15/2020 09:34:31", "content": "

Thank you for replying me! My network CNN has 12 M parameters and I use batch size is 16. It runs in about 30 mins per epoch. Training full train set causes faults and I'm stucking with it

", "rawMarkdown": "Thank you for replying me! My network CNN has 12 M parameters and I use batch size is 16. It runs in about 30 mins per epoch. Training full train set causes faults and I'm stucking with it", "votes": null }, { "id": "719259", "postDate": "01/15/2020 10:13:12", "content": "

First thing first ,don’t use TQDM especially notebook with colab and chrome . That’s suicide. It will give JavaScript error., hang chrome and complete system after reconnect .

\n\n

Second things , small batch size , less complex models , clearing gradient etc etc.

", "rawMarkdown": "First thing first ,don’t use TQDM especially notebook with colab and chrome . That’s suicide. It will give JavaScript error., hang chrome and complete system after reconnect .\n\nSecond things , small batch size , less complex models , clearing gradient etc etc.", "votes": null }, { "id": "719361", "postDate": "01/15/2020 12:27:44", "content": "

Another thing to keep in mind is removing excessive prints to the console. For e.g. try to silently unzip the files, otherwise it might crash your session.

", "rawMarkdown": "Another thing to keep in mind is removing excessive prints to the console. For e.g. try to silently unzip the files, otherwise it might crash your session.", "votes": null }, { "id": "719369", "postDate": "01/15/2020 12:42:18", "content": "

I tried remove the tqdm and now my notebook runs without crashing. Thanks you so much!

", "rawMarkdown": "I tried remove the tqdm and now my notebook runs without crashing. Thanks you so much!", "votes": null }, { "id": "719372", "postDate": "01/15/2020 12:43:35", "content": "

I have fixed by removing tqdm like @phoenix9032 suggestion and it work. Thank you for suggestion!

", "rawMarkdown": "I have fixed by removing tqdm like @phoenix9032 suggestion and it work. Thank you for suggestion!", "votes": null }, { "id": "720276", "postDate": "01/16/2020 09:39:50", "content": "

I see that you have solved your issue ! Even so, I suggest you to consult this discussion where Uday Kamal gave a very piece of JS code to be put in the browser console in order to prevent a tab containing a Colab execution to disconnect with time !

\n\n

javascript\nfunction ClickConnect(){\nconsole.log(\"Working\"); \ndocument.querySelector(\"colab-toolbar-button#connect\").click() \n}\nsetInterval(ClickConnect,60000)\n\nSource

", "rawMarkdown": "I see that you have solved your issue ! Even so, I suggest you to consult [this discussion](https://www.kaggle.com/c/bengaliai-cv19/discussion/124646#711322) where Uday Kamal gave a very piece of JS code to be put in the browser console in order to prevent a tab containing a Colab execution to disconnect with time !\n\n```javascript\nfunction ClickConnect(){\nconsole.log(\"Working\"); \ndocument.querySelector(\"colab-toolbar-button#connect\").click() \n}\nsetInterval(ClickConnect,60000)\n```\n[Source](https://medium.com/@shivamrawat_756/how-to-prevent-google-colab-from-disconnecting-717b88a128c0)", "votes": null }, { "id": "720512", "postDate": "01/16/2020 13:44:07", "content": "

How did you manage to move the huge dataset to Gdrive?

", "rawMarkdown": "How did you manage to move the huge dataset to Gdrive?", "votes": null }, { "id": "720515", "postDate": "01/16/2020 13:46:50", "content": "

I am using this https://www.kaggle.com/iafoss/grapheme-imgs-128x128 dataset. I am unable to extract the zip file as the notebook and chrome stop responding. Any suggestions on what to do?

", "rawMarkdown": "I am using this https://www.kaggle.com/iafoss/grapheme-imgs-128x128 dataset. I am unable to extract the zip file as the notebook and chrome stop responding. Any suggestions on what to do?", "votes": null }, { "id": "720702", "postDate": "01/16/2020 16:49:47", "content": "

@venky2506 Sorry for double message, I could not edit this first one without it being truncated by Kaggle...

", "rawMarkdown": "venky2506 Sorry for double message, I could not edit this first one without it being truncated by Kaggle...", "votes": null }, { "id": "720711", "postDate": "01/16/2020 16:57:59", "content": "

@venky2506 Personally, I zipped it and then access it this way:

\n\n

```\nimport zipfile\nzip_ref = zipfile.ZipFile(path_to_file, 'r')

\n\n

def get_np_array_from_zip_ref(zip_file):\n # converts a buffer from a zip file in np.array\n return np.asarray(\n bytearray(zip_file.read())\n , dtype=np.uint8)

\n\n

img = cv2.imdecode(\n get_np_array_from_zip_ref(zip_ref.open(image_filename)),\n 0\n)

\n\n

```

", "rawMarkdown": "venky2506 Personally, I zipped it and then access it this way:\n\n```\nimport zipfile\nzip_ref = zipfile.ZipFile(path_to_file, 'r')\n\ndef get_np_array_from_zip_ref(zip_file):\n # converts a buffer from a zip file in np.array\n return np.asarray(\n bytearray(zip_file.read())\n , dtype=np.uint8)\n\nimg = cv2.imdecode(\n get_np_array_from_zip_ref(zip_ref.open(image_filename)),\n 0\n)\n\n```", "votes": null }, { "id": "720757", "postDate": "01/16/2020 17:46:37", "content": "

!unzip something.zip > /dev/null

", "rawMarkdown": "!unzip something.zip > /dev/null", "votes": null }, { "id": "721822", "postDate": "01/17/2020 18:24:59", "content": "

I am doing that. But i suppose due to copious amounts of images, the notebook crashes.

", "rawMarkdown": "I am doing that. But i suppose due to copious amounts of images, the notebook crashes.", "votes": null }, { "id": "722393", "postDate": "01/18/2020 14:06:50", "content": "

Thank you for your suggestion. I will try it (y)

", "rawMarkdown": "Thank you for your suggestion. I will try it (y)", "votes": null }, { "id": "722476", "postDate": "01/18/2020 16:08:17", "content": "

Try paperspace @linhlpv

", "rawMarkdown": "Try paperspace @linhlpv", "votes": null }, { "id": "723729", "postDate": "01/20/2020 11:41:03", "content": "

Thanks for your suggestion! I have used both paperspace and colab but I think paperspace seem quite difficult to use. But it is a usefull resource for all kagglers :3

", "rawMarkdown": "Thanks for your suggestion! I have used both paperspace and colab but I think paperspace seem quite difficult to use. But it is a usefull resource for all kagglers :3", "votes": null }, { "id": "727067", "postDate": "01/23/2020 12:37:32", "content": "

There’s a pretty nice chrome extension that helps re-connects you. It works on Chromium-based browers too, I’ve used it on Brave a few times.

", "rawMarkdown": "There’s a pretty nice [chrome extension](https://chrome.google.com/webstore/detail/colab-auto-reconnect/nbcihfbfamjlfiopdcemmohoojdecjid) that helps re-connects you. It works on Chromium-based browers too, I’ve used it on Brave a few times.", "votes": null }, { "id": "878168", "postDate": "06/08/2020 10:13:41", "content": "

I have training a network in tensorflow 2.0 the ram used is continuously increasing, then sessions are crashed both in colab and as kaggle notebook. can someone suggest a reason ?

", "rawMarkdown": "I have training a network in tensorflow 2.0 the ram used is continuously increasing, then sessions are crashed both in colab and as kaggle notebook. can someone suggest a reason ?", "votes": null }, { "id": "911321", "postDate": "07/01/2020 16:53:42", "content": "

Try Q blocks. If you want an easy to use interface with jupyter notebooks and 1/10th the cost for GPU instances.

", "rawMarkdown": "Try Q blocks. If you want an easy to use interface with jupyter notebooks and 1/10th the cost for GPU instances.", "votes": null } ], "comments": [ { "id": 719240, "author_name": "imeintanis", "author_url": "", "post_date": "01/15/2020 09:30:00", "content": "

Make sure you use GPU enabled runtime. Apart from that try with\n- smaller batches\n- smaller image size\n- simpler network\nand increase progressively to see how it performs.

\n\n

From my experience a CNN net with about Trainable params: 18,000,000, image_size=128x128 and batch_size=64 runs in about 20 mins per epoch.

", "votes": null, "replies": [ { "id": 719243, "author_name": "linhlpv", "author_url": "", "post_date": "01/15/2020 09:34:31", "content": "

Thank you for replying me! My network CNN has 12 M parameters and I use batch size is 16. It runs in about 30 mins per epoch. Training full train set causes faults and I'm stucking with it

", "votes": null, "replies": [] } ] }, { "id": 719259, "author_name": "phoenix9032", "author_url": "", "post_date": "01/15/2020 10:13:12", "content": "

First thing first ,don’t use TQDM especially notebook with colab and chrome . That’s suicide. It will give JavaScript error., hang chrome and complete system after reconnect .

\n\n

Second things , small batch size , less complex models , clearing gradient etc etc.

", "votes": null, "replies": [ { "id": 719369, "author_name": "linhlpv", "author_url": "", "post_date": "01/15/2020 12:42:18", "content": "

I tried remove the tqdm and now my notebook runs without crashing. Thanks you so much!

", "votes": null, "replies": [] } ] }, { "id": 719361, "author_name": "sumeetdhariwal", "author_url": "", "post_date": "01/15/2020 12:27:44", "content": "

Another thing to keep in mind is removing excessive prints to the console. For e.g. try to silently unzip the files, otherwise it might crash your session.

", "votes": null, "replies": [ { "id": 719372, "author_name": "linhlpv", "author_url": "", "post_date": "01/15/2020 12:43:35", "content": "

I have fixed by removing tqdm like @phoenix9032 suggestion and it work. Thank you for suggestion!

", "votes": null, "replies": [] } ] }, { "id": 720276, "author_name": "dimartinot", "author_url": "", "post_date": "01/16/2020 09:39:50", "content": "

I see that you have solved your issue ! Even so, I suggest you to consult this discussion where Uday Kamal gave a very piece of JS code to be put in the browser console in order to prevent a tab containing a Colab execution to disconnect with time !

\n\n

javascript\nfunction ClickConnect(){\nconsole.log(\"Working\"); \ndocument.querySelector(\"colab-toolbar-button#connect\").click() \n}\nsetInterval(ClickConnect,60000)\n\nSource

", "votes": null, "replies": [ { "id": 722393, "author_name": "linhlpv", "author_url": "", "post_date": "01/18/2020 14:06:50", "content": "

Thank you for your suggestion. I will try it (y)

", "votes": null, "replies": [] } ] }, { "id": 720512, "author_name": "venky2506", "author_url": "", "post_date": "01/16/2020 13:44:07", "content": "

How did you manage to move the huge dataset to Gdrive?

", "votes": null, "replies": [ { "id": 720702, "author_name": "dimartinot", "author_url": "", "post_date": "01/16/2020 16:49:47", "content": "

@venky2506 Sorry for double message, I could not edit this first one without it being truncated by Kaggle...

", "votes": null, "replies": [] }, { "id": 720711, "author_name": "dimartinot", "author_url": "", "post_date": "01/16/2020 16:57:59", "content": "

@venky2506 Personally, I zipped it and then access it this way:

\n\n

```\nimport zipfile\nzip_ref = zipfile.ZipFile(path_to_file, 'r')

\n\n

def get_np_array_from_zip_ref(zip_file):\n # converts a buffer from a zip file in np.array\n return np.asarray(\n bytearray(zip_file.read())\n , dtype=np.uint8)

\n\n

img = cv2.imdecode(\n get_np_array_from_zip_ref(zip_ref.open(image_filename)),\n 0\n)

\n\n

```

", "votes": null, "replies": [] } ] }, { "id": 720515, "author_name": "venky2506", "author_url": "", "post_date": "01/16/2020 13:46:50", "content": "

I am using this https://www.kaggle.com/iafoss/grapheme-imgs-128x128 dataset. I am unable to extract the zip file as the notebook and chrome stop responding. Any suggestions on what to do?

", "votes": null, "replies": [ { "id": 720757, "author_name": "timetraveller98", "author_url": "", "post_date": "01/16/2020 17:46:37", "content": "

!unzip something.zip > /dev/null

", "votes": null, "replies": [] }, { "id": 721822, "author_name": "venky2506", "author_url": "", "post_date": "01/17/2020 18:24:59", "content": "

I am doing that. But i suppose due to copious amounts of images, the notebook crashes.

", "votes": null, "replies": [] } ] }, { "id": 722476, "author_name": "kurianbenoy", "author_url": "", "post_date": "01/18/2020 16:08:17", "content": "

Try paperspace @linhlpv

", "votes": null, "replies": [ { "id": 723729, "author_name": "linhlpv", "author_url": "", "post_date": "01/20/2020 11:41:03", "content": "

Thanks for your suggestion! I have used both paperspace and colab but I think paperspace seem quite difficult to use. But it is a usefull resource for all kagglers :3

", "votes": null, "replies": [] }, { "id": 911321, "author_name": "genesis96839", "author_url": "", "post_date": "07/01/2020 16:53:42", "content": "

Try Q blocks. If you want an easy to use interface with jupyter notebooks and 1/10th the cost for GPU instances.

", "votes": null, "replies": [] } ] }, { "id": 727067, "author_name": "maxlenormand", "author_url": "", "post_date": "01/23/2020 12:37:32", "content": "

There’s a pretty nice chrome extension that helps re-connects you. It works on Chromium-based browers too, I’ve used it on Brave a few times.

", "votes": null, "replies": [] }, { "id": 878168, "author_name": "moulicm111", "author_url": "", "post_date": "06/08/2020 10:13:41", "content": "

I have training a network in tensorflow 2.0 the ram used is continuously increasing, then sessions are crashed both in colab and as kaggle notebook. can someone suggest a reason ?

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "719186": "Because limited quota of kaggle, I move to use Colab for training my model. Everything seems ok when I trained small data, but when I train all of the data, the Colab have been disconnected after 1 epoch and trying reconnect caused my Chrome crash (my ram increased very fast and I did not know why). Did anybody have the same issue and How could I do to solve this problems. Thank you all!", "719240": "Make sure you use GPU enabled runtime. Apart from that try with\n- smaller batches\n- smaller image size\n- simpler network\nand increase progressively to see how it performs. \n\nFrom my experience a CNN net with about `Trainable params: 18,000,000`, `image_size=128x128` and `batch_size=64` runs in about 20 mins per epoch.", "719243": "Thank you for replying me! My network CNN has 12 M parameters and I use batch size is 16. It runs in about 30 mins per epoch. Training full train set causes faults and I'm stucking with it", "719259": "First thing first ,don’t use TQDM especially notebook with colab and chrome . That’s suicide. It will give JavaScript error., hang chrome and complete system after reconnect .\n\nSecond things , small batch size , less complex models , clearing gradient etc etc.", "719361": "Another thing to keep in mind is removing excessive prints to the console. For e.g. try to silently unzip the files, otherwise it might crash your session.", "719369": "I tried remove the tqdm and now my notebook runs without crashing. Thanks you so much!", "719372": "I have fixed by removing tqdm like @phoenix9032 suggestion and it work. Thank you for suggestion!", "720276": "I see that you have solved your issue ! Even so, I suggest you to consult [this discussion](https://www.kaggle.com/c/bengaliai-cv19/discussion/124646#711322) where Uday Kamal gave a very piece of JS code to be put in the browser console in order to prevent a tab containing a Colab execution to disconnect with time !\n\n```javascript\nfunction ClickConnect(){\nconsole.log(\"Working\"); \ndocument.querySelector(\"colab-toolbar-button#connect\").click() \n}\nsetInterval(ClickConnect,60000)\n```\n[Source](https://medium.com/@shivamrawat_756/how-to-prevent-google-colab-from-disconnecting-717b88a128c0)", "720512": "How did you manage to move the huge dataset to Gdrive?", "720515": "I am using this https://www.kaggle.com/iafoss/grapheme-imgs-128x128 dataset. I am unable to extract the zip file as the notebook and chrome stop responding. Any suggestions on what to do?", "720702": "venky2506 Sorry for double message, I could not edit this first one without it being truncated by Kaggle...", "720711": "venky2506 Personally, I zipped it and then access it this way:\n\n```\nimport zipfile\nzip_ref = zipfile.ZipFile(path_to_file, 'r')\n\ndef get_np_array_from_zip_ref(zip_file):\n # converts a buffer from a zip file in np.array\n return np.asarray(\n bytearray(zip_file.read())\n , dtype=np.uint8)\n\nimg = cv2.imdecode(\n get_np_array_from_zip_ref(zip_ref.open(image_filename)),\n 0\n)\n\n```", "720757": "!unzip something.zip > /dev/null", "721822": "I am doing that. But i suppose due to copious amounts of images, the notebook crashes.", "722393": "Thank you for your suggestion. I will try it (y)", "722476": "Try paperspace @linhlpv", "723729": "Thanks for your suggestion! I have used both paperspace and colab but I think paperspace seem quite difficult to use. But it is a usefull resource for all kagglers :3", "727067": "There’s a pretty nice [chrome extension](https://chrome.google.com/webstore/detail/colab-auto-reconnect/nbcihfbfamjlfiopdcemmohoojdecjid) that helps re-connects you. It works on Chromium-based browers too, I’ve used it on Brave a few times.", "878168": "I have training a network in tensorflow 2.0 the ram used is continuously increasing, then sessions are crashed both in colab and as kaggle notebook. can someone suggest a reason ?", "911321": "Try Q blocks. If you want an easy to use interface with jupyter notebooks and 1/10th the cost for GPU instances." }, "source": "meta" }