When I run my code on kaggle notebook, the kernel dies due to exceed of RAM.
\nis there anyway to do it locally and upload jupyter file only? (for public)
or is there anyway to interact with google cloud instance?
", "messages": [ { "id": "2133611", "postDate": "02/07/2023 14:18:14", "content": "When I run my code on kaggle notebook, the kernel dies due to exceed of RAM.
\nis there anyway to do it locally and upload jupyter file only? (for public)
or is there anyway to interact with google cloud instance?
", "rawMarkdown": "When I run my code on kaggle notebook, the kernel dies due to exceed of RAM.\nis there anyway to do it locally and upload jupyter file only? (for public)\n\nor is there anyway to interact with google cloud instance?", "votes": null }, { "id": "2133704", "postDate": "02/07/2023 15:04:01", "content": "For example, you can train your model locally and then upload the trained model file to a Kaggle Dataset.
", "rawMarkdown": "For example, you can train your model locally and then upload the trained model file to a Kaggle Dataset.", "votes": null }, { "id": "2133737", "postDate": "02/07/2023 15:18:27", "content": "You can try use the GPU and instead of pandas use CuDF which works on the GPU memory and is much faster than Pandas.
", "rawMarkdown": "You can try use the GPU and instead of pandas use CuDF which works on the GPU memory and is much faster than Pandas.", "votes": null }, { "id": "2133756", "postDate": "02/07/2023 15:27:21", "content": "You can not use GPU in this competition:
\n\n", "rawMarkdown": "You can not use GPU in this competition: \n>Note: this competition is aimed at producing models that are small and lightweight. We have introduced compute constraints to match - your VMs will have only 2 CPUs, 8GB of RAM, and no GPU available.", "votes": null }, { "id": "2133769", "postDate": "02/07/2023 15:33:32", "content": "Note: this competition is aimed at producing models that are small and lightweight. We have introduced compute constraints to match - your VMs will have only 2 CPUs, 8GB of RAM, and no GPU available.
\n
You can do EDA or model training on other environment such as GCP instance or google colab notebook, but you should run your code on kaggle notebook when you submit your \"submission.csv\" because this is code competition. And you cannot interact with other environments when submitting your prediction because internet-off setting is needed.
\nIf you want to save RAM, there are some solutions. If you use pandas.read_csv, you can save RAM usage by limiting number of rows to be loaded on memory by setting nrows , setting appropriate (more memory saving) dtype explicitly, or selecting columns to be loaded on memory by usecols, for example. Using other dataframe library, such as dask, polars, sometimes save more RAM than pandas.
Sorry, I thought this was for only the efficiency prize.
", "rawMarkdown": "Sorry, I thought this was for only the efficiency prize.", "votes": null }, { "id": "2135505", "postDate": "02/08/2023 17:15:17", "content": "understood. thanks!
", "rawMarkdown": "understood. thanks!", "votes": null }, { "id": "2135507", "postDate": "02/08/2023 17:15:39", "content": "yes, I will try that…thank for the help.
", "rawMarkdown": "yes, I will try that...thank for the help.", "votes": null } ], "comments": [ { "id": 2133704, "author_name": "allvor", "author_url": "", "post_date": "02/07/2023 15:04:01", "content": "For example, you can train your model locally and then upload the trained model file to a Kaggle Dataset.
", "votes": null, "replies": [ { "id": 2135507, "author_name": "kimtaehun", "author_url": "", "post_date": "02/08/2023 17:15:39", "content": "yes, I will try that…thank for the help.
", "votes": null, "replies": [] } ] }, { "id": 2133737, "author_name": "mohammad2012191", "author_url": "", "post_date": "02/07/2023 15:18:27", "content": "You can try use the GPU and instead of pandas use CuDF which works on the GPU memory and is much faster than Pandas.
", "votes": null, "replies": [ { "id": 2133756, "author_name": "allvor", "author_url": "", "post_date": "02/07/2023 15:27:21", "content": "You can not use GPU in this competition:
\n\n", "votes": null, "replies": [ { "id": 2133785, "author_name": "mohammad2012191", "author_url": "", "post_date": "02/07/2023 15:44:08", "content": "Note: this competition is aimed at producing models that are small and lightweight. We have introduced compute constraints to match - your VMs will have only 2 CPUs, 8GB of RAM, and no GPU available.
\n
Sorry, I thought this was for only the efficiency prize.
", "votes": null, "replies": [] } ] } ] }, { "id": 2133769, "author_name": "tomokikmogura", "author_url": "", "post_date": "02/07/2023 15:33:32", "content": "You can do EDA or model training on other environment such as GCP instance or google colab notebook, but you should run your code on kaggle notebook when you submit your \"submission.csv\" because this is code competition. And you cannot interact with other environments when submitting your prediction because internet-off setting is needed.
\nIf you want to save RAM, there are some solutions. If you use pandas.read_csv, you can save RAM usage by limiting number of rows to be loaded on memory by setting nrows , setting appropriate (more memory saving) dtype explicitly, or selecting columns to be loaded on memory by usecols, for example. Using other dataframe library, such as dask, polars, sometimes save more RAM than pandas.
understood. thanks!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2133611": "When I run my code on kaggle notebook, the kernel dies due to exceed of RAM.\nis there anyway to do it locally and upload jupyter file only? (for public)\n\nor is there anyway to interact with google cloud instance?", "2133704": "For example, you can train your model locally and then upload the trained model file to a Kaggle Dataset.", "2133737": "You can try use the GPU and instead of pandas use CuDF which works on the GPU memory and is much faster than Pandas.", "2133756": "You can not use GPU in this competition: \n>Note: this competition is aimed at producing models that are small and lightweight. We have introduced compute constraints to match - your VMs will have only 2 CPUs, 8GB of RAM, and no GPU available.", "2133769": "You can do EDA or model training on other environment such as GCP instance or google colab notebook, but you should run your code on kaggle notebook when you submit your \"submission.csv\" because this is code competition. And you cannot interact with other environments when submitting your prediction because internet-off setting is needed. \n\nIf you want to save RAM, there are some solutions. If you use [`pandas.read_csv`](https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html), you can save RAM usage by limiting number of rows to be loaded on memory by setting `nrows` , setting appropriate (more memory saving) `dtype` explicitly, or selecting columns to be loaded on memory by `usecols`, for example. Using other dataframe library, such as [`dask`](https://www.dask.org/or), [`polars`](https://www.pola.rs/), sometimes save more RAM than pandas.", "2133785": "Sorry, I thought this was for only the efficiency prize.", "2135505": "understood. thanks!", "2135507": "yes, I will try that...thank for the help." }, "source": "meta" }