When I load all data and try run to some simple analysis, I often ran out of memory. So, a window pops up and ask me to upgrade to google cloud AI notebook. Are you in this \"normal\" notebook or google cloud AI notebook with maybe much more memory? Does it cost to upgrade?
", "messages": [ { "id": "1094712", "postDate": "11/28/2020 21:03:17", "content": "When I load all data and try run to some simple analysis, I often ran out of memory. So, a window pops up and ask me to upgrade to google cloud AI notebook. Are you in this \"normal\" notebook or google cloud AI notebook with maybe much more memory? Does it cost to upgrade?
", "rawMarkdown": "When I load all data and try run to some simple analysis, I often ran out of memory. So, a window pops up and ask me to upgrade to google cloud AI notebook. Are you in this \"normal\" notebook or google cloud AI notebook with maybe much more memory? Does it cost to upgrade?", "votes": null }, { "id": "1095098", "postDate": "11/29/2020 08:39:42", "content": "I'm using colab and its 25GB of RAM. I've got three kernels in which I compute specific features (ex : kernel 1 => question features, kernel 2 : lecture features…) and save to drive the training and validation sets. Then I've got a kernel dedicated to training in which I load the computed features and perform the training. In this way I can bypass the memory issues for free (for the moment). Could be interesting to move to cloud services for the last weeks in order to train over the entire dataset IMHO
", "rawMarkdown": "I'm using colab and its 25GB of RAM. I've got three kernels in which I compute specific features (ex : kernel 1 => question features, kernel 2 : lecture features...) and save to drive the training and validation sets. Then I've got a kernel dedicated to training in which I load the computed features and perform the training. In this way I can bypass the memory issues for free (for the moment). Could be interesting to move to cloud services for the last weeks in order to train over the entire dataset IMHO", "votes": null }, { "id": "1095633", "postDate": "11/29/2020 19:49:02", "content": "Good to know. The competition has CPU, GPU requirement, but not memory limitation. By the way, do you need to pay for upgrading?
", "rawMarkdown": "Good to know. The competition has CPU, GPU requirement, but not memory limitation. By the way, do you need to pay for upgrading?", "votes": null }, { "id": "1095969", "postDate": "11/30/2020 05:25:52", "content": "Yes! It is definitely a paid feature. You have to set up the billing. I would suggest training your model on 10% of the data to avoid memory error.
\nIf you are running out of memory during simple analysis, it is mostly because you have read the CSV directly without fixing the data types. Public Notebooks in this competition have a lot of hints on reading with less memory usage. Do follow them.
\nGood Luck!
", "rawMarkdown": "Yes! It is definitely a paid feature. You have to set up the billing. I would suggest training your model on 10% of the data to avoid memory error.\n\nIf you are running out of memory during simple analysis, it is mostly because you have read the CSV directly without fixing the data types. Public Notebooks in this competition have a lot of hints on reading with less memory usage. Do follow them.\n\nGood Luck!", "votes": null }, { "id": "1096577", "postDate": "11/30/2020 15:59:56", "content": "\n\nI'm using colab and its 25GB of RAM
\n
How do you get that? I thought they removed the feature of doubling the RAM…
\nIs it Colab Pro you are using?
yes, colab pro allows 25GB RAM
", "rawMarkdown": "yes, colab pro allows 25GB RAM", "votes": null }, { "id": "1097795", "postDate": "12/01/2020 08:46:20", "content": "I used to work with colab pro but I've canceled my subscription some months ago. I've still got 25GB of RAM, I thought it was also the case for free version
", "rawMarkdown": "I used to work with colab pro but I've canceled my subscription some months ago. I've still got 25GB of RAM, I thought it was also the case for free version", "votes": null }, { "id": "1100340", "postDate": "12/03/2020 02:46:54", "content": "Hey, I want to ask if I upgrade to colab notebook, can kaggle directly run colab notebook to submit. Or I must convert it to kaggle notebook first and then commit.
", "rawMarkdown": "Hey, I want to ask if I upgrade to colab notebook, can kaggle directly run colab notebook to submit. Or I must convert it to kaggle notebook first and then commit.", "votes": null }, { "id": "1100824", "postDate": "12/03/2020 11:40:38", "content": "Colab/Colab pro - you will have to convert to kaggle notebooks (you can upload notebook though) to commit.
", "rawMarkdown": "Colab/Colab pro - you will have to convert to kaggle notebooks (you can upload notebook though) to commit.", "votes": null }, { "id": "1101581", "postDate": "12/04/2020 03:29:51", "content": "Thanks a lot.
", "rawMarkdown": "Thanks a lot.", "votes": null }, { "id": "1104271", "postDate": "12/06/2020 19:06:55", "content": "Anyone knows how to easily access google drive files from google cloud AI notebook ? It could be useful to train the model over the entire dataset without RAM issues
", "rawMarkdown": "Anyone knows how to easily access google drive files from google cloud AI notebook ? It could be useful to train the model over the entire dataset without RAM issues", "votes": null } ], "comments": [ { "id": 1095098, "author_name": "alexj21", "author_url": "", "post_date": "11/29/2020 08:39:42", "content": "I'm using colab and its 25GB of RAM. I've got three kernels in which I compute specific features (ex : kernel 1 => question features, kernel 2 : lecture features…) and save to drive the training and validation sets. Then I've got a kernel dedicated to training in which I load the computed features and perform the training. In this way I can bypass the memory issues for free (for the moment). Could be interesting to move to cloud services for the last weeks in order to train over the entire dataset IMHO
", "votes": null, "replies": [ { "id": 1095633, "author_name": "noahxi", "author_url": "", "post_date": "11/29/2020 19:49:02", "content": "Good to know. The competition has CPU, GPU requirement, but not memory limitation. By the way, do you need to pay for upgrading?
", "votes": null, "replies": [] }, { "id": 1095969, "author_name": "lplenka", "author_url": "", "post_date": "11/30/2020 05:25:52", "content": "Yes! It is definitely a paid feature. You have to set up the billing. I would suggest training your model on 10% of the data to avoid memory error.
\nIf you are running out of memory during simple analysis, it is mostly because you have read the CSV directly without fixing the data types. Public Notebooks in this competition have a lot of hints on reading with less memory usage. Do follow them.
\nGood Luck!
", "votes": null, "replies": [] }, { "id": 1096577, "author_name": "stecasasso", "author_url": "", "post_date": "11/30/2020 15:59:56", "content": "\n\nI'm using colab and its 25GB of RAM
\n
How do you get that? I thought they removed the feature of doubling the RAM…
\nIs it Colab Pro you are using?
yes, colab pro allows 25GB RAM
", "votes": null, "replies": [] }, { "id": 1097795, "author_name": "alexj21", "author_url": "", "post_date": "12/01/2020 08:46:20", "content": "I used to work with colab pro but I've canceled my subscription some months ago. I've still got 25GB of RAM, I thought it was also the case for free version
", "votes": null, "replies": [] } ] }, { "id": 1100340, "author_name": "weikangqiu", "author_url": "", "post_date": "12/03/2020 02:46:54", "content": "Hey, I want to ask if I upgrade to colab notebook, can kaggle directly run colab notebook to submit. Or I must convert it to kaggle notebook first and then commit.
", "votes": null, "replies": [ { "id": 1100824, "author_name": "rashmibanthia", "author_url": "", "post_date": "12/03/2020 11:40:38", "content": "Colab/Colab pro - you will have to convert to kaggle notebooks (you can upload notebook though) to commit.
", "votes": null, "replies": [] }, { "id": 1101581, "author_name": "weikangqiu", "author_url": "", "post_date": "12/04/2020 03:29:51", "content": "Thanks a lot.
", "votes": null, "replies": [] } ] }, { "id": 1104271, "author_name": "alexj21", "author_url": "", "post_date": "12/06/2020 19:06:55", "content": "Anyone knows how to easily access google drive files from google cloud AI notebook ? It could be useful to train the model over the entire dataset without RAM issues
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1094712": "When I load all data and try run to some simple analysis, I often ran out of memory. So, a window pops up and ask me to upgrade to google cloud AI notebook. Are you in this \"normal\" notebook or google cloud AI notebook with maybe much more memory? Does it cost to upgrade?", "1095098": "I'm using colab and its 25GB of RAM. I've got three kernels in which I compute specific features (ex : kernel 1 => question features, kernel 2 : lecture features...) and save to drive the training and validation sets. Then I've got a kernel dedicated to training in which I load the computed features and perform the training. In this way I can bypass the memory issues for free (for the moment). Could be interesting to move to cloud services for the last weeks in order to train over the entire dataset IMHO", "1095633": "Good to know. The competition has CPU, GPU requirement, but not memory limitation. By the way, do you need to pay for upgrading?", "1095969": "Yes! It is definitely a paid feature. You have to set up the billing. I would suggest training your model on 10% of the data to avoid memory error.\n\nIf you are running out of memory during simple analysis, it is mostly because you have read the CSV directly without fixing the data types. Public Notebooks in this competition have a lot of hints on reading with less memory usage. Do follow them.\n\nGood Luck!", "1096577": "> I'm using colab and its 25GB of RAM\n\nHow do you get that? I thought they removed the feature of doubling the RAM...\nIs it Colab Pro you are using?", "1097480": "yes, colab pro allows 25GB RAM", "1097795": "I used to work with colab pro but I've canceled my subscription some months ago. I've still got 25GB of RAM, I thought it was also the case for free version", "1100340": "Hey, I want to ask if I upgrade to colab notebook, can kaggle directly run colab notebook to submit. Or I must convert it to kaggle notebook first and then commit.", "1100824": "Colab/Colab pro - you will have to convert to kaggle notebooks (you can upload notebook though) to commit.", "1101581": "Thanks a lot.", "1104271": "Anyone knows how to easily access google drive files from google cloud AI notebook ? It could be useful to train the model over the entire dataset without RAM issues" }, "source": "meta" }