One of the main challenges I faced when working on this competition is the 30 hours per week GPU quota limit in Kaggle. I experimented with using GoogleColab, it was not usable to me because of the lack of background session capability, lack of input data persistence between sessions, google drive mounting/copying too slow, CPU memory lower than Kaggle on the free version etc. I also tried the “Upgrade to Google Cloud AI notebooks” and it turns out the $300 free credits are not usable for GPU instances, and the GPU instances equivalent to Kaggle runtime were priced at around $500 per month. I saw that @hengck23 shared in a notebook that they were using the following which is in a whole different league than Kaggle resources.
\nZ8-G4 Data Science Workstation
\nNow that I read the 1st place solution by @clevert, I realized that there is no way that all kidneys training at that resolution will fit in a Kaggle runtime. Some other 3D training solutions posted would also not fit in Kaggle runtime.
\nI am curious to know how many teams here, especially medal winning ones, used Kaggle CPU/GPU exclusively for training, cross validation and inference? Please raise your hand by replying to this post. Special Kudos to those fellow teams!
\nAlso, to those who used another more powerful workstation, would you be willing to share the specs of here?
", "messages": [ { "id": 2643652, "postDate": "2024-02-09T01:59:01.440Z", "content": "wow, everyone has high gpu computation and memory!
\nHow a newbie self-study datascience plan is changing:
\nyear 2012:
\nyear 2014:
\nyear 2018:
\nyear 2022:
\nyear 2024:
\nthough uncommon, but not rare.
\ni do receive a couple of emails, expressing interest in sponsoring GPU, tools etc … from time to time
if you check website, you can also find invitation (for being ambassador ) via write in for a couple of products.
\ni would suggest why don't you just write in and try. but before that, give good proposal (why should people sponsor you a gpu or cloud gpu) … it is good if you have large followers in youtube, tiktok, kaggler, blogs, linkedin, face-to-face/virtual meetup etc becuase these are good for marketing, etc
", "rawMarkdown": "though uncommon, but not rare.\ni do receive a couple of emails, expressing interest in sponsoring GPU, tools etc ... from time to time\n\nif you check website, you can also find invitation (for being ambassador ) via write in for a couple of products.\n\ni would suggest why don't you just write in and try. but before that, give good proposal (why should people sponsor you a gpu or cloud gpu) ... it is good if you have large followers in youtube, tiktok, kaggler, blogs, linkedin, face-to-face/virtual meetup etc becuase these are good for marketing, etc", "votes": 1 }, { "id": 2643670, "postDate": "2024-02-09T02:35:00.280Z", "content": "just take this competition as an example.
\nin short, if you want to to be very sucessful in kaggle, you need powerful resources, particularly gpu. you have to find them. finding resources is part of the competition
", "rawMarkdown": "just take this competition as an example.\n\n1. start a discord group and ask people to signup\n2. what you want to do is to create a study group so that each of the members can modify their code to get same results as the top. \n3. in the study group we share code, experiment results, discussion ... do post submisisons, more ablation study, presnetation, etc ...\n4. if you do this over time you can get followers ... and then ....\n\n\n---\n\nin short, if you want to to be very sucessful in kaggle, you need powerful resources, particularly gpu. you have to find them. finding resources is part of the competition\n " }, { "id": 2643672, "postDate": "2024-02-09T02:35:09.597Z", "rawMarkdown": "", "isDeleted": true } ] } ] }, { "id": 2643543, "postDate": "2024-02-08T22:22:27.890Z", "content": "One of the main challenges I faced when working on this competition is the 30 hours per week GPU quota limit in Kaggle. I experimented with using GoogleColab, it was not usable to me because of the lack of background session capability, lack of input data persistence between sessions, google drive mounting/copying too slow, CPU memory lower than Kaggle on the free version etc. I also tried the “Upgrade to Google Cloud AI notebooks” and it turns out the $300 free credits are not usable for GPU instances, and the GPU instances equivalent to Kaggle runtime were priced at around $500 per month. I saw that @hengck23 shared in a notebook that they were using the following which is in a whole different league than Kaggle resources.
\nZ8-G4 Data Science Workstation
\nNow that I read the 1st place solution by @clevert, I realized that there is no way that all kidneys training at that resolution will fit in a Kaggle runtime. Some other 3D training solutions posted would also not fit in Kaggle runtime.
\nI am curious to know how many teams here, especially medal winning ones, used Kaggle CPU/GPU exclusively for training, cross validation and inference? Please raise your hand by replying to this post. Special Kudos to those fellow teams!
\nAlso, to those who used another more powerful workstation, would you be willing to share the specs of here?
", "rawMarkdown": "One of the main challenges I faced when working on this competition is the 30 hours per week GPU quota limit in Kaggle. I experimented with using GoogleColab, it was not usable to me because of the lack of background session capability, lack of input data persistence between sessions, google drive mounting/copying too slow, CPU memory lower than Kaggle on the free version etc. I also tried the “Upgrade to Google Cloud AI notebooks” and it turns out the $300 free credits are not usable for GPU instances, and the GPU instances equivalent to Kaggle runtime were priced at around $500 per month. I saw that @hengck23 shared in a notebook that they were using the following which is in a whole different league than Kaggle resources.\n\nZ8-G4 Data Science Workstation\n* GPU: 2x Nvidia Quadro RTX 8000, each with VRAM 48 GB\n* CPU: Intel® Xeon(R) Gold 6240 CPU @ 2.60GHz, 72 cores\n* Memory: 376 GB RAM\n\nNow that I read the 1st place solution by @clevert, I realized that there is no way that all kidneys training at that resolution will fit in a Kaggle runtime. Some other 3D training solutions posted would also not fit in Kaggle runtime.\n\nI am curious to know how many teams here, especially medal winning ones, used Kaggle CPU/GPU exclusively for training, cross validation and inference? Please raise your hand by replying to this post. Special Kudos to those fellow teams!\n\nAlso, to those who used another more powerful workstation, would you be willing to share the specs of here?", "votes": 3 }, { "id": 2643649, "postDate": "2024-02-09T01:43:30.847Z", "content": "I was actually able to get away with training on 40GB of gpu this time around but I only trained with 2 kidney types.
", "rawMarkdown": "I was actually able to get away with training on 40GB of gpu this time around but I only trained with 2 kidney types. ", "votes": 1 }, { "id": 2643612, "postDate": "2024-02-09T00:37:00.737Z", "content": "I used an rtx 3060 (12 gb vram) for most of this competition. I could implement most ideas, just needed to keep batch size small or number of filters low.
", "rawMarkdown": "I used an rtx 3060 (12 gb vram) for most of this competition. I could implement most ideas, just needed to keep batch size small or number of filters low.", "votes": 1 }, { "id": 2643631, "postDate": "2024-02-09T01:20:30.890Z", "content": "1x4090, 80GB CPU RAM. I also used laptop 4060 for code writing and code testing short runs.
", "rawMarkdown": "1x4090, 80GB CPU RAM. I also used laptop 4060 for code writing and code testing short runs. ", "votes": 2 }, { "id": 2643597, "postDate": "2024-02-09T00:08:27.803Z", "content": "\"Now that I read the 1st place solution by @clevert, I realized that there is no way that all kidneys training at that resolution will fit in a Kaggle runtime.\"
\nactaully if your model did not have batch norm (e.g. repalce with layer norm or group norm), you can use gradient accumation:
\n batch train data\n gradient\n image batch:\n loss = net( image)\n compute gradient ( accumulate ) backward() \n backprop \nthis is how i train my 3d for crop size 256x256x256
", "rawMarkdown": "\"Now that I read the 1st place solution by @clevert, I realized that there is no way that all kidneys training at that resolution will fit in a Kaggle runtime.\"\n\n\nactaully if your model did not have batch norm (e.g. repalce with layer norm or group norm), you can use gradient accumation:\n\n```\nfor each batch in train data\n zero gradient\n for each image in batch:\n loss = net(one image)\n compute gradient (and accumulate ) using backward() # (loss/batch_size).backward()\n do backprop # optimizer.step() \n\n```\n\nthis is how i train my 3d for crop size 256x256x256", "votes": 2, "replies": [ { "id": 2643611, "postDate": "2024-02-09T00:35:40.083Z", "content": "The tip about later norm or group norm is a good one. I want to learn more about those, as well as gradient accumulation, next time. Another thing relevant to OP's post is mixed precision training, which is also on my list.
", "rawMarkdown": "The tip about later norm or group norm is a good one. I want to learn more about those, as well as gradient accumulation, next time. Another thing relevant to OP's post is mixed precision training, which is also on my list.", "replies": [ { "id": 2643627, "postDate": "2024-02-09T01:05:03.950Z", "content": "the gradient accumation is not enough, you still gradient checkpointing
\nnow kaggle gpu has about 16 GB, with gradient checkpointing, you will have virtually x2 (32 GB) of train vram, but slower training iteration.
https://github.com/prigoyal/pytorch_memonger/blob/master/tutorial/Checkpointing_for_PyTorch_models.ipynb
\nhttps://huggingface.co/docs/transformers/v4.20.1/en/perf_train_gpu_one
\nhttps://huggingface.co/docs/transformers/v4.18.0/en/performance
Thanks for the references @hengck23 ! I will read up on this :)
", "rawMarkdown": "Thanks for the references @hengck23 ! I will read up on this :)" } ] } ] }, { "id": 2643549, "postDate": "2024-02-08T22:26:54.273Z", "content": "I used Kaggle GPUs for trainings only in my 1st competition on Kaggle. Back then you could get up to 6 instances that were running for (I think) 9 hours straight.
\nQuickly after that the quotas were reduced, and I had to move to my personal hardware. It was always lacking in power, but in the past year managed to build a devbox with 2xA6000 Ada. So 96GB of VRAM and (almost) speed of RTX 4090. Finally, I could implement most of ideas in competitions.
", "rawMarkdown": "I used Kaggle GPUs for trainings only in my 1st competition on Kaggle. Back then you could get up to 6 instances that were running for (I think) 9 hours straight. \n\nQuickly after that the quotas were reduced, and I had to move to my personal hardware. It was always lacking in power, but in the past year managed to build a devbox with 2xA6000 Ada. So 96GB of VRAM and (almost) speed of RTX 4090. Finally, I could implement most of ideas in competitions. ", "votes": 2 } ], "comments": [ { "id": 2643652, "author_name": "hengck23", "author_url": "", "post_date": "2024-02-09T01:59:01.440000", "content": "wow, everyone has high gpu computation and memory!
\nHow a newbie self-study datascience plan is changing:
\nyear 2012:
\nyear 2014:
\nyear 2018:
\nyear 2022:
\nyear 2024:
\nthough uncommon, but not rare.
\ni do receive a couple of emails, expressing interest in sponsoring GPU, tools etc … from time to time
if you check website, you can also find invitation (for being ambassador ) via write in for a couple of products.
\ni would suggest why don't you just write in and try. but before that, give good proposal (why should people sponsor you a gpu or cloud gpu) … it is good if you have large followers in youtube, tiktok, kaggler, blogs, linkedin, face-to-face/virtual meetup etc becuase these are good for marketing, etc
", "votes": 1, "replies": [] }, { "id": 2643670, "author_name": "hengck23", "author_url": "", "post_date": "2024-02-09T02:35:00.280000", "content": "just take this competition as an example.
\nin short, if you want to to be very sucessful in kaggle, you need powerful resources, particularly gpu. you have to find them. finding resources is part of the competition
", "votes": 0, "replies": [] }, { "id": 2643672, "author_name": "", "author_url": "", "post_date": "2024-02-09T02:35:09.597000", "content": "", "votes": 0, "replies": [] } ] } ] }, { "id": 2643649, "author_name": "Cody_Null", "author_url": "", "post_date": "2024-02-09T01:43:30.847000", "content": "I was actually able to get away with training on 40GB of gpu this time around but I only trained with 2 kidney types.
", "votes": 1, "replies": [] }, { "id": 2643612, "author_name": "chemdatafarmer", "author_url": "", "post_date": "2024-02-09T00:37:00.737000", "content": "I used an rtx 3060 (12 gb vram) for most of this competition. I could implement most ideas, just needed to keep batch size small or number of filters low.
", "votes": 1, "replies": [] }, { "id": 2643631, "author_name": "joejeo1", "author_url": "", "post_date": "2024-02-09T01:20:30.890000", "content": "1x4090, 80GB CPU RAM. I also used laptop 4060 for code writing and code testing short runs.
", "votes": 2, "replies": [] }, { "id": 2643597, "author_name": "hengck23", "author_url": "", "post_date": "2024-02-09T00:08:27.803000", "content": "\"Now that I read the 1st place solution by @clevert, I realized that there is no way that all kidneys training at that resolution will fit in a Kaggle runtime.\"
\nactaully if your model did not have batch norm (e.g. repalce with layer norm or group norm), you can use gradient accumation:
\n batch train data\n gradient\n image batch:\n loss = net( image)\n compute gradient ( accumulate ) backward() \n backprop \nthis is how i train my 3d for crop size 256x256x256
", "votes": 2, "replies": [ { "id": 2643611, "author_name": "chemdatafarmer", "author_url": "", "post_date": "2024-02-09T00:35:40.083000", "content": "The tip about later norm or group norm is a good one. I want to learn more about those, as well as gradient accumulation, next time. Another thing relevant to OP's post is mixed precision training, which is also on my list.
", "votes": 0, "replies": [ { "id": 2643627, "author_name": "hengck23", "author_url": "", "post_date": "2024-02-09T01:05:03.950000", "content": "the gradient accumation is not enough, you still gradient checkpointing
\nnow kaggle gpu has about 16 GB, with gradient checkpointing, you will have virtually x2 (32 GB) of train vram, but slower training iteration.
https://github.com/prigoyal/pytorch_memonger/blob/master/tutorial/Checkpointing_for_PyTorch_models.ipynb
\nhttps://huggingface.co/docs/transformers/v4.20.1/en/perf_train_gpu_one
\nhttps://huggingface.co/docs/transformers/v4.18.0/en/performance
Thanks for the references @hengck23 ! I will read up on this :)
", "votes": 0, "replies": [] } ] } ] }, { "id": 2643549, "author_name": "Ivan Panshin", "author_url": "", "post_date": "2024-02-08T22:26:54.273000", "content": "I used Kaggle GPUs for trainings only in my 1st competition on Kaggle. Back then you could get up to 6 instances that were running for (I think) 9 hours straight.
\nQuickly after that the quotas were reduced, and I had to move to my personal hardware. It was always lacking in power, but in the past year managed to build a devbox with 2xA6000 Ada. So 96GB of VRAM and (almost) speed of RTX 4090. Finally, I could implement most of ideas in competitions.
", "votes": 2, "replies": [] } ], "raw_markdown_by_id": { "2643652": "wow, everyone has high gpu computation and memory!\n\nHow a newbie self-study datascience plan is changing:\n\nyear 2012:\n- read paper \n- participate in kaggle\n\nyear 2014:\n- read paper \n- read github \n- learn online course \n- win a bronze in kaggle\n\nyear 2018:\n- read paper \n- read github \n- learn online course\n- setup various autotune tools, train framework tools \n- win a silver in kaggle\n\nyear 2022:\n- read paper \n- read github \n- learn online course\n- setup various autotune tools, train framework tools + accelerate/distribution training framework \n- setup cloud gpu for scaleup training\n- earn money or find sponsor to buy gpu machine\n- learn from discord, etc\n- win a gold in kaggle\n\n\nyear 2024:\n- read paper \n- read github \n- learn online course\n- setup various autotune tools, train framework tools\n- setup cloud gpu for scaleup training\n- earn money or find sponsor to buy gpu machine\n- learn from discord, etc\n- how to ask chatgpt for help\n- foundation models to label more data?\n- top 3 in kaggle\n", "2643543": "One of the main challenges I faced when working on this competition is the 30 hours per week GPU quota limit in Kaggle. I experimented with using GoogleColab, it was not usable to me because of the lack of background session capability, lack of input data persistence between sessions, google drive mounting/copying too slow, CPU memory lower than Kaggle on the free version etc. I also tried the “Upgrade to Google Cloud AI notebooks” and it turns out the $300 free credits are not usable for GPU instances, and the GPU instances equivalent to Kaggle runtime were priced at around $500 per month. I saw that @hengck23 shared in a notebook that they were using the following which is in a whole different league than Kaggle resources.\n\nZ8-G4 Data Science Workstation\n* GPU: 2x Nvidia Quadro RTX 8000, each with VRAM 48 GB\n* CPU: Intel® Xeon(R) Gold 6240 CPU @ 2.60GHz, 72 cores\n* Memory: 376 GB RAM\n\nNow that I read the 1st place solution by @clevert, I realized that there is no way that all kidneys training at that resolution will fit in a Kaggle runtime. Some other 3D training solutions posted would also not fit in Kaggle runtime.\n\nI am curious to know how many teams here, especially medal winning ones, used Kaggle CPU/GPU exclusively for training, cross validation and inference? Please raise your hand by replying to this post. Special Kudos to those fellow teams!\n\nAlso, to those who used another more powerful workstation, would you be willing to share the specs of here?", "2643649": "I was actually able to get away with training on 40GB of gpu this time around but I only trained with 2 kidney types. ", "2643612": "I used an rtx 3060 (12 gb vram) for most of this competition. I could implement most ideas, just needed to keep batch size small or number of filters low.", "2643631": "1x4090, 80GB CPU RAM. I also used laptop 4060 for code writing and code testing short runs. ", "2643597": "\"Now that I read the 1st place solution by @clevert, I realized that there is no way that all kidneys training at that resolution will fit in a Kaggle runtime.\"\n\n\nactaully if your model did not have batch norm (e.g. repalce with layer norm or group norm), you can use gradient accumation:\n\n```\nfor each batch in train data\n zero gradient\n for each image in batch:\n loss = net(one image)\n compute gradient (and accumulate ) using backward() # (loss/batch_size).backward()\n do backprop # optimizer.step() \n\n```\n\nthis is how i train my 3d for crop size 256x256x256", "2643549": "I used Kaggle GPUs for trainings only in my 1st competition on Kaggle. Back then you could get up to 6 instances that were running for (I think) 9 hours straight. \n\nQuickly after that the quotas were reduced, and I had to move to my personal hardware. It was always lacking in power, but in the past year managed to build a devbox with 2xA6000 Ada. So 96GB of VRAM and (almost) speed of RTX 4090. Finally, I could implement most of ideas in competitions. " } }