{ "id": 295782, "title": "Advice for low computing resources", "url": "/competitions/tensorflow-great-barrier-reef/discussion/295782", "author_name": "", "post_date": "2021-12-17T20:14:27.425260400Z", "votes": 8, "comment_count": 8, "views": 0, "content": "

Hey guys,
\nI'm fairly new to image processing. I have low computing power resources. I was wondering if google Colab Notebook is enough to handle big datasets. Is 24gB of ram enough for a project of this amplitude? How do you efficiently manage the memory ?

\n

Thank you

", "messages": [ { "id": "1621552", "postDate": "12/17/2021 20:14:27", "content": "

Hey guys,
\nI'm fairly new to image processing. I have low computing power resources. I was wondering if google Colab Notebook is enough to handle big datasets. Is 24gB of ram enough for a project of this amplitude? How do you efficiently manage the memory ?

\n

Thank you

", "rawMarkdown": "Hey guys,\nI'm fairly new to image processing. I have low computing power resources. I was wondering if google Colab Notebook is enough to handle big datasets. Is 24gB of ram enough for a project of this amplitude? How do you efficiently manage the memory ?\n\nThank you", "votes": null }, { "id": "1622121", "postDate": "12/18/2021 11:36:41", "content": "

We prepare all our models on Google Colab Pro+ (due to background feautre). I am sure Colab Pro will be enough. Unfortunately there are many limitations …. slow file system (gdrive) and GPU 16GB (and mostyly P100) … which impact your training time significantly … (my yolo epoch training time is … about 35 minutes) ….

", "rawMarkdown": "We prepare all our models on Google Colab Pro+ (due to background feautre). I am sure Colab Pro will be enough. Unfortunately there are many limitations .... slow file system (gdrive) and GPU 16GB (and mostyly P100) ... which impact your training time significantly ... (my yolo epoch training time is ... about 35 minutes) ....", "votes": null }, { "id": "1622337", "postDate": "12/18/2021 16:09:25", "content": "

I kept getting runs kicked out of the free Colab when training. You may need to try a paid (pro, pro+) version of Colab & hope you still don't get kicked out for too long a training time. You can also use private Kaggle notebooks with a 40 hr/week GPU limit. Remek noted 35 minute epoch times on Colab - my recollection is Kaggle gave me 5-6 or so epochs per hour. RAM is not so much the issue, 24 G likely fine, GPU memory matters more.

", "rawMarkdown": "I kept getting runs kicked out of the free Colab when training. You may need to try a paid (pro, pro+) version of Colab & hope you still don't get kicked out for too long a training time. You can also use private Kaggle notebooks with a 40 hr/week GPU limit. Remek noted 35 minute epoch times on Colab - my recollection is Kaggle gave me 5-6 or so epochs per hour. RAM is not so much the issue, 24 G likely fine, GPU memory matters more.", "votes": null }, { "id": "1622342", "postDate": "12/18/2021 16:15:43", "content": "

I use Colab Pro+ …. 24/h …. no problem and backgroud feature is really nice. You can just turn off your computer … and go slip … morning notebook is still working. You are absolutely right … for this dataset RAM is not problem …. GPU is problem … 16GB is LOW … very LOW … (batch size 4-10 depends on image res). It is big limitation … because it limit us from agile experimenting ….

", "rawMarkdown": "I use Colab Pro+ .... 24/h .... no problem and backgroud feature is really nice. You can just turn off your computer ... and go slip ... morning notebook is still working. You are absolutely right ... for this dataset RAM is not problem .... GPU is problem ... 16GB is LOW ... very LOW ... (batch size 4-10 depends on image res). It is big limitation ... because it limit us from agile experimenting ....", "votes": null }, { "id": "1622569", "postDate": "12/18/2021 22:37:54", "content": "

Hello @julienwitty, I also recommend Colab Pro, I think the ratio of price to performance is good.

", "rawMarkdown": "Hello @julienwitty, I also recommend Colab Pro, I think the ratio of price to performance is good.", "votes": null }, { "id": "1624813", "postDate": "12/21/2021 09:00:30", "content": "

I'm using Colab Pro for this project. There are several issues.

\n
    \n
  1. yes, it takes about 15 mins to copy the subset of data I am using from my Google drive to the colab drive each time I run it.
    2. \n
  2. yes, it can time out, but that's usually after 12 hrs and I'm not running this notebook anywhere near that long.
    3. \n
  3. You can lose your results if it disconnects after it finishes and before you get back to it. This is easily solved by copying the results to my Google drive at the end of the script.
    4. \n
\n

As for epoch time… that will depend on how you are breaking up the data. I'm not finding it an issue.

", "rawMarkdown": "I'm using Colab Pro for this project. There are several issues. \n1. yes, it takes about 15 mins to copy the subset of data I am using from my Google drive to the colab drive each time I run it.\n2. yes, it can time out, but that's usually after 12 hrs and I'm not running this notebook anywhere near that long.\n3. You can lose your results if it disconnects after it finishes and before you get back to it. This is easily solved by copying the results to my Google drive at the end of the script.\n\nAs for epoch time... that will depend on how you are breaking up the data. I'm not finding it an issue.", "votes": null }, { "id": "1624896", "postDate": "12/21/2021 10:51:24", "content": "

ad.3. Pro+ has background feature … even you are offline notebook works … it is cool feature. Many times I lost connection … then when I restarted notebook it works (do not stop).

", "rawMarkdown": "ad.3. Pro+ has background feature ... even you are offline notebook works ... it is cool feature. Many times I lost connection ... then when I restarted notebook it works (do not stop).", "votes": null }, { "id": "1625848", "postDate": "12/22/2021 07:39:28", "content": "

Thank's for the replies, everyone. I used colab pro in the past for school projects. I was asking this because I didn't want to invest a lot of time in a project where I don't have the computing resources.

\n

Thank's for the insight guys. Good luck for the rest of the competition :)

", "rawMarkdown": "Thank's for the replies, everyone. I used colab pro in the past for school projects. I was asking this because I didn't want to invest a lot of time in a project where I don't have the computing resources.\n\nThank's for the insight guys. Good luck for the rest of the competition :)", "votes": null }, { "id": "1626191", "postDate": "12/22/2021 14:54:16", "content": "

There are options where you can get GPU power from online and you can use it on computer. Other than that you can use google Colab

", "rawMarkdown": "There are options where you can get GPU power from online and you can use it on computer. Other than that you can use google Colab", "votes": null } ], "comments": [ { "id": 1622121, "author_name": "remekkinas", "author_url": "", "post_date": "12/18/2021 11:36:41", "content": "

We prepare all our models on Google Colab Pro+ (due to background feautre). I am sure Colab Pro will be enough. Unfortunately there are many limitations …. slow file system (gdrive) and GPU 16GB (and mostyly P100) … which impact your training time significantly … (my yolo epoch training time is … about 35 minutes) ….

", "votes": null, "replies": [] }, { "id": 1622337, "author_name": "vettejeep", "author_url": "", "post_date": "12/18/2021 16:09:25", "content": "

I kept getting runs kicked out of the free Colab when training. You may need to try a paid (pro, pro+) version of Colab & hope you still don't get kicked out for too long a training time. You can also use private Kaggle notebooks with a 40 hr/week GPU limit. Remek noted 35 minute epoch times on Colab - my recollection is Kaggle gave me 5-6 or so epochs per hour. RAM is not so much the issue, 24 G likely fine, GPU memory matters more.

", "votes": null, "replies": [ { "id": 1622342, "author_name": "remekkinas", "author_url": "", "post_date": "12/18/2021 16:15:43", "content": "

I use Colab Pro+ …. 24/h …. no problem and backgroud feature is really nice. You can just turn off your computer … and go slip … morning notebook is still working. You are absolutely right … for this dataset RAM is not problem …. GPU is problem … 16GB is LOW … very LOW … (batch size 4-10 depends on image res). It is big limitation … because it limit us from agile experimenting ….

", "votes": null, "replies": [] } ] }, { "id": 1622569, "author_name": "cv13j0", "author_url": "", "post_date": "12/18/2021 22:37:54", "content": "

Hello @julienwitty, I also recommend Colab Pro, I think the ratio of price to performance is good.

", "votes": null, "replies": [] }, { "id": 1624813, "author_name": "mutantspore", "author_url": "", "post_date": "12/21/2021 09:00:30", "content": "

I'm using Colab Pro for this project. There are several issues.

\n
    \n
  1. yes, it takes about 15 mins to copy the subset of data I am using from my Google drive to the colab drive each time I run it.
    2. \n
  2. yes, it can time out, but that's usually after 12 hrs and I'm not running this notebook anywhere near that long.
    3. \n
  3. You can lose your results if it disconnects after it finishes and before you get back to it. This is easily solved by copying the results to my Google drive at the end of the script.
    4. \n
\n

As for epoch time… that will depend on how you are breaking up the data. I'm not finding it an issue.

", "votes": null, "replies": [ { "id": 1624896, "author_name": "remekkinas", "author_url": "", "post_date": "12/21/2021 10:51:24", "content": "

ad.3. Pro+ has background feature … even you are offline notebook works … it is cool feature. Many times I lost connection … then when I restarted notebook it works (do not stop).

", "votes": null, "replies": [] } ] }, { "id": 1625848, "author_name": "julienwitty", "author_url": "", "post_date": "12/22/2021 07:39:28", "content": "

Thank's for the replies, everyone. I used colab pro in the past for school projects. I was asking this because I didn't want to invest a lot of time in a project where I don't have the computing resources.

\n

Thank's for the insight guys. Good luck for the rest of the competition :)

", "votes": null, "replies": [] }, { "id": 1626191, "author_name": "karthikbhandary2", "author_url": "", "post_date": "12/22/2021 14:54:16", "content": "

There are options where you can get GPU power from online and you can use it on computer. Other than that you can use google Colab

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1621552": "Hey guys,\nI'm fairly new to image processing. I have low computing power resources. I was wondering if google Colab Notebook is enough to handle big datasets. Is 24gB of ram enough for a project of this amplitude? How do you efficiently manage the memory ?\n\nThank you", "1622121": "We prepare all our models on Google Colab Pro+ (due to background feautre). I am sure Colab Pro will be enough. Unfortunately there are many limitations .... slow file system (gdrive) and GPU 16GB (and mostyly P100) ... which impact your training time significantly ... (my yolo epoch training time is ... about 35 minutes) ....", "1622337": "I kept getting runs kicked out of the free Colab when training. You may need to try a paid (pro, pro+) version of Colab & hope you still don't get kicked out for too long a training time. You can also use private Kaggle notebooks with a 40 hr/week GPU limit. Remek noted 35 minute epoch times on Colab - my recollection is Kaggle gave me 5-6 or so epochs per hour. RAM is not so much the issue, 24 G likely fine, GPU memory matters more.", "1622342": "I use Colab Pro+ .... 24/h .... no problem and backgroud feature is really nice. You can just turn off your computer ... and go slip ... morning notebook is still working. You are absolutely right ... for this dataset RAM is not problem .... GPU is problem ... 16GB is LOW ... very LOW ... (batch size 4-10 depends on image res). It is big limitation ... because it limit us from agile experimenting ....", "1622569": "Hello @julienwitty, I also recommend Colab Pro, I think the ratio of price to performance is good.", "1624813": "I'm using Colab Pro for this project. There are several issues. \n1. yes, it takes about 15 mins to copy the subset of data I am using from my Google drive to the colab drive each time I run it.\n2. yes, it can time out, but that's usually after 12 hrs and I'm not running this notebook anywhere near that long.\n3. You can lose your results if it disconnects after it finishes and before you get back to it. This is easily solved by copying the results to my Google drive at the end of the script.\n\nAs for epoch time... that will depend on how you are breaking up the data. I'm not finding it an issue.", "1624896": "ad.3. Pro+ has background feature ... even you are offline notebook works ... it is cool feature. Many times I lost connection ... then when I restarted notebook it works (do not stop).", "1625848": "Thank's for the replies, everyone. I used colab pro in the past for school projects. I was asking this because I didn't want to invest a lot of time in a project where I don't have the computing resources.\n\nThank's for the insight guys. Good luck for the rest of the competition :)", "1626191": "There are options where you can get GPU power from online and you can use it on computer. Other than that you can use google Colab" }, "source": "meta" }