I was considering this competition to be my first one on Kaggle. However, even for a classification task competition, it seems to require a good hardware. So, I was wondering if anyone could share any thoughts on what is the minimum hardware requirement to be competitive. Is the Kaggle and Colab infrastructure enough? If you don't think there is a good answer for this question, could you share your hardware settings so I can get a sense of what do I need to be start competing on Kaggle? Any comment/help is welcome!
\n\nThanks in advance
", "messages": [ { "id": "882097", "postDate": "06/11/2020 15:21:30", "content": "I was considering this competition to be my first one on Kaggle. However, even for a classification task competition, it seems to require a good hardware. So, I was wondering if anyone could share any thoughts on what is the minimum hardware requirement to be competitive. Is the Kaggle and Colab infrastructure enough? If you don't think there is a good answer for this question, could you share your hardware settings so I can get a sense of what do I need to be start competing on Kaggle? Any comment/help is welcome!
\n\nThanks in advance
", "rawMarkdown": "I was considering this competition to be my first one on Kaggle. However, even for a classification task competition, it seems to require a good hardware. So, I was wondering if anyone could share any thoughts on what is the minimum hardware requirement to be competitive. Is the Kaggle and Colab infrastructure enough? If you don't think there is a good answer for this question, could you share your hardware settings so I can get a sense of what do I need to be start competing on Kaggle? Any comment/help is welcome! \n\nThanks in advance", "votes": null }, { "id": "882119", "postDate": "06/11/2020 15:41:25", "content": "In previous Image competitions, top participants typically had access to 32GB RAM machines with 2 GTX/RTX GPUs.
\n\nHowever, now that we have TPU option in addition to GPUs on Kaggle/Colab, and also 2 months until the deadline, perhaps you can use techniques such as load/save intermediate checkpoints, and just use the free Kaggle/Colab environments for training your models.
", "rawMarkdown": "In previous Image competitions, top participants typically had access to 32GB RAM machines with 2 GTX/RTX GPUs.\n\nHowever, now that we have TPU option in addition to GPUs on Kaggle/Colab, and also 2 months until the deadline, perhaps you can use techniques such as load/save intermediate checkpoints, and just use the free Kaggle/Colab environments for training your models.", "votes": null }, { "id": "882137", "postDate": "06/11/2020 15:55:18", "content": "It pretty impressive what kaggle and colab gives you for free. \nkaggle gives you 10 concurrent ssesions with 4 CPU cores/16GB each as long as you want\nplus 30h/week P100 GPU :-)\nplus 30h/week TPU :-)))
\n\nTry the TPUs. You can not buy a system for your living room beeing so fast for little money ;-)
\n\nIf you want a local system, because using a IDE for development instead of the notebooks, a better pc should be enough for most competitions. 4-12 core, 16-32GB RAM, 1080 Ti or RTX 2070Super.
\n\nIn competitions try to start with downsampled/partial data. Implement/finetune your models/code/workflow.\nThen scale up the models, data and use TPUs.
", "rawMarkdown": "It pretty impressive what kaggle and colab gives you for free. \nkaggle gives you 10 concurrent ssesions with 4 CPU cores/16GB each as long as you want\nplus 30h/week P100 GPU :-)\nplus 30h/week TPU :-)))\n\nTry the TPUs. You can not buy a system for your living room beeing so fast for little money ;-)\n\nIf you want a local system, because using a IDE for development instead of the notebooks, a better pc should be enough for most competitions. 4-12 core, 16-32GB RAM, 1080 Ti or RTX 2070Super.\n\nIn competitions try to start with downsampled/partial data. Implement/finetune your models/code/workflow.\nThen scale up the models, data and use TPUs.", "votes": null }, { "id": "882146", "postDate": "06/11/2020 16:04:17", "content": "Nice! Thank you so much! What about disk space? On this competition, for instance, the data size is bigger than 100GB, is this a problem for Kaggle or Colab?
\n\nThanks again!
", "rawMarkdown": "Nice! Thank you so much! What about disk space? On this competition, for instance, the data size is bigger than 100GB, is this a problem for Kaggle or Colab?\n\nThanks again!", "votes": null }, { "id": "882169", "postDate": "06/11/2020 16:18:21", "content": "A notebook can use up to 5GB diskspace as output or temporary read/write. This is not very much. You have 16GB RAM but only 5GB disk... but you can have multiple notebooks and each can produce 5GB of results, which are stored permanently. \nThe input-data does not count to this 5GB limit, and each notebook can use multiple dataset as input and also the outputs of multiple other notebooks. You can chain the notebooks.
\n\nA second possibility is to create datasets (public or private). -> from Link\n- 20GB per dataset limit\n- 20GB max private datasets (if you exceed this, either make your datasets public or delete unused datasets)\n- A max of 50 top-level files (if you have more, use a directory structure and upload an archive)
", "rawMarkdown": "A notebook can use up to 5GB diskspace as output or temporary read/write. This is not very much. You have 16GB RAM but only 5GB disk... but you can have multiple notebooks and each can produce 5GB of results, which are stored permanently. \nThe input-data does not count to this 5GB limit, and each notebook can use multiple dataset as input and also the outputs of multiple other notebooks. You can chain the notebooks.\n\nA second possibility is to create datasets (public or private). -> from [Link](https://www.kaggle.com/docs/datasets)\n- 20GB per dataset limit\n- 20GB max private datasets (if you exceed this, either make your datasets public or delete unused datasets)\n- A max of 50 top-level files (if you have more, use a directory structure and upload an archive)", "votes": null }, { "id": "882226", "postDate": "06/11/2020 16:47:55", "content": "Thank you very much! I think that's all I need to know to get started! =))
", "rawMarkdown": "Thank you very much! I think that's all I need to know to get started! =))", "votes": null } ], "comments": [ { "id": 882119, "author_name": "sirishks", "author_url": "", "post_date": "06/11/2020 15:41:25", "content": "In previous Image competitions, top participants typically had access to 32GB RAM machines with 2 GTX/RTX GPUs.
\n\nHowever, now that we have TPU option in addition to GPUs on Kaggle/Colab, and also 2 months until the deadline, perhaps you can use techniques such as load/save intermediate checkpoints, and just use the free Kaggle/Colab environments for training your models.
", "votes": null, "replies": [] }, { "id": 882137, "author_name": "agentauers", "author_url": "", "post_date": "06/11/2020 15:55:18", "content": "It pretty impressive what kaggle and colab gives you for free. \nkaggle gives you 10 concurrent ssesions with 4 CPU cores/16GB each as long as you want\nplus 30h/week P100 GPU :-)\nplus 30h/week TPU :-)))
\n\nTry the TPUs. You can not buy a system for your living room beeing so fast for little money ;-)
\n\nIf you want a local system, because using a IDE for development instead of the notebooks, a better pc should be enough for most competitions. 4-12 core, 16-32GB RAM, 1080 Ti or RTX 2070Super.
\n\nIn competitions try to start with downsampled/partial data. Implement/finetune your models/code/workflow.\nThen scale up the models, data and use TPUs.
", "votes": null, "replies": [ { "id": 882146, "author_name": "bpmsilva", "author_url": "", "post_date": "06/11/2020 16:04:17", "content": "Nice! Thank you so much! What about disk space? On this competition, for instance, the data size is bigger than 100GB, is this a problem for Kaggle or Colab?
\n\nThanks again!
", "votes": null, "replies": [] }, { "id": 882169, "author_name": "agentauers", "author_url": "", "post_date": "06/11/2020 16:18:21", "content": "A notebook can use up to 5GB diskspace as output or temporary read/write. This is not very much. You have 16GB RAM but only 5GB disk... but you can have multiple notebooks and each can produce 5GB of results, which are stored permanently. \nThe input-data does not count to this 5GB limit, and each notebook can use multiple dataset as input and also the outputs of multiple other notebooks. You can chain the notebooks.
\n\nA second possibility is to create datasets (public or private). -> from Link\n- 20GB per dataset limit\n- 20GB max private datasets (if you exceed this, either make your datasets public or delete unused datasets)\n- A max of 50 top-level files (if you have more, use a directory structure and upload an archive)
", "votes": null, "replies": [] }, { "id": 882226, "author_name": "bpmsilva", "author_url": "", "post_date": "06/11/2020 16:47:55", "content": "Thank you very much! I think that's all I need to know to get started! =))
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "882097": "I was considering this competition to be my first one on Kaggle. However, even for a classification task competition, it seems to require a good hardware. So, I was wondering if anyone could share any thoughts on what is the minimum hardware requirement to be competitive. Is the Kaggle and Colab infrastructure enough? If you don't think there is a good answer for this question, could you share your hardware settings so I can get a sense of what do I need to be start competing on Kaggle? Any comment/help is welcome! \n\nThanks in advance", "882119": "In previous Image competitions, top participants typically had access to 32GB RAM machines with 2 GTX/RTX GPUs.\n\nHowever, now that we have TPU option in addition to GPUs on Kaggle/Colab, and also 2 months until the deadline, perhaps you can use techniques such as load/save intermediate checkpoints, and just use the free Kaggle/Colab environments for training your models.", "882137": "It pretty impressive what kaggle and colab gives you for free. \nkaggle gives you 10 concurrent ssesions with 4 CPU cores/16GB each as long as you want\nplus 30h/week P100 GPU :-)\nplus 30h/week TPU :-)))\n\nTry the TPUs. You can not buy a system for your living room beeing so fast for little money ;-)\n\nIf you want a local system, because using a IDE for development instead of the notebooks, a better pc should be enough for most competitions. 4-12 core, 16-32GB RAM, 1080 Ti or RTX 2070Super.\n\nIn competitions try to start with downsampled/partial data. Implement/finetune your models/code/workflow.\nThen scale up the models, data and use TPUs.", "882146": "Nice! Thank you so much! What about disk space? On this competition, for instance, the data size is bigger than 100GB, is this a problem for Kaggle or Colab?\n\nThanks again!", "882169": "A notebook can use up to 5GB diskspace as output or temporary read/write. This is not very much. You have 16GB RAM but only 5GB disk... but you can have multiple notebooks and each can produce 5GB of results, which are stored permanently. \nThe input-data does not count to this 5GB limit, and each notebook can use multiple dataset as input and also the outputs of multiple other notebooks. You can chain the notebooks.\n\nA second possibility is to create datasets (public or private). -> from [Link](https://www.kaggle.com/docs/datasets)\n- 20GB per dataset limit\n- 20GB max private datasets (if you exceed this, either make your datasets public or delete unused datasets)\n- A max of 50 top-level files (if you have more, use a directory structure and upload an archive)", "882226": "Thank you very much! I think that's all I need to know to get started! =))" }, "source": "meta" }