{ "id": 222953, "title": "Compute power limitation is so frustrating ^^", "url": "/competitions/ranzcr-clip-catheter-line-classification/discussion/222953", "author_name": "", "post_date": "2021-03-01T20:27:40.687996700Z", "votes": 25, "comment_count": 18, "views": 0, "content": "

The idea I am trying right now is showing promising results but I am forced to use relatively low resolutions given I am limited by compute power .
\nI am 100% sure I would have much better results with higher resolutions and train faster with better compute power. It's a bit frustrating^^

", "messages": [ { "id": "1222480", "postDate": "03/01/2021 20:27:40", "content": "

The idea I am trying right now is showing promising results but I am forced to use relatively low resolutions given I am limited by compute power .
\nI am 100% sure I would have much better results with higher resolutions and train faster with better compute power. It's a bit frustrating^^

", "rawMarkdown": "The idea I am trying right now is showing promising results but I am forced to use relatively low resolutions given I am limited by compute power . \nI am 100% sure I would have much better results with higher resolutions and train faster with better compute power. It's a bit frustrating^^", "votes": null }, { "id": "1222526", "postDate": "03/01/2021 21:33:46", "content": "

Yeah, more is always better. I saw some people mention 1024x1024 images and that is pretty much completely out of the question for me and I would consider myself reasonably well equipped.

", "rawMarkdown": "Yeah, more is always better. I saw some people mention 1024x1024 images and that is pretty much completely out of the question for me and I would consider myself reasonably well equipped.", "votes": null }, { "id": "1222539", "postDate": "03/01/2021 21:56:56", "content": "

The first and unique time I could train with 1024x1024 was using TFRec and Kaggle TPU v3 with Bfloat16

\n

But here I use Pytorch and V100 16 Go

", "rawMarkdown": "The first and unique time I could train with 1024x1024 was using TFRec and Kaggle TPU v3 with Bfloat16\n\nBut here I use Pytorch and V100 16 Go", "votes": null }, { "id": "1222589", "postDate": "03/01/2021 23:45:35", "content": "

That's why kagglers with less compute power focuses on ensemble most of the cases.
\nI dislike gpu war :(
\nI thought as we have gn,frn,ws these days
\nIt will be easy to compete with kagglers having good gpu
\nBut the image size becomes the bottleneck here again
\nI tried to train resnet34 for long time on little bit bigger image size but it Doesn't work.
\n\"Bigger is better\" :(
\nI Forcefully use tensorflow sometimes even though i am pytorch lover
\nWhy i use tensorflow?
\nV3-8 tpu or tpu pod access
\nThat's crazy fast
\nBut i am not very comfortable with tensorflow
\nI wish torch xla had similar power like tf tpu :(

", "rawMarkdown": "That's why kagglers with less compute power focuses on ensemble most of the cases.\nI dislike gpu war :(\nI thought as we have gn,frn,ws these days\nIt will be easy to compete with kagglers having good gpu\nBut the image size becomes the bottleneck here again\nI tried to train resnet34 for long time on little bit bigger image size but it Doesn't work.\n\"Bigger is better\" :(\nI Forcefully use tensorflow sometimes even though i am pytorch lover\nWhy i use tensorflow? \nV3-8 tpu or tpu pod access\nThat's crazy fast\nBut i am not very comfortable with tensorflow \nI wish torch xla had similar power like tf tpu :(", "votes": null }, { "id": "1222861", "postDate": "03/02/2021 08:18:07", "content": "

The images size really matters here.
\nAnd my current approach is power greedy. But unfortunately I can't go further than 416x416. And I am already using Gradient Accumulation and all sort of optimizations.

", "rawMarkdown": "The images size really matters here.\nAnd my current approach is power greedy. But unfortunately I can't go further than 416x416. And I am already using Gradient Accumulation and all sort of optimizations.", "votes": null }, { "id": "1222871", "postDate": "03/02/2021 08:28:19", "content": "

agree,today from my one experiment i just realized that more than 512 img size is better, i don't know if you can really get good result with 416 img size,it will be difficult to get 0.955 with this image size but probably still possible with deeper networks

", "rawMarkdown": "agree,today from my one experiment i just realized that more than 512 img size is better, i don't know if you can really get good result with 416 img size,it will be difficult to get 0.955 with this image size but probably still possible with deeper networks", "votes": null }, { "id": "1222897", "postDate": "03/02/2021 09:11:24", "content": "

I feel for you brother… I an also having a hard time in this competition due to lack of processing power… This is the first time I am feeling this helpless… I almost drain all of my GPU and TPU hours with 1-2 days remaining and them I have to wait for Saturday 😅
\nBut it is what it is… I am just happy for the learning opportunity and discovering what it could have been with bigger image size…

", "rawMarkdown": "I feel for you brother... I an also having a hard time in this competition due to lack of processing power... This is the first time I am feeling this helpless... I almost drain all of my GPU and TPU hours with 1-2 days remaining and them I have to wait for Saturday 😅\nBut it is what it is... I am just happy for the learning opportunity and discovering what it could have been with bigger image size...", "votes": null }, { "id": "1222930", "postDate": "03/02/2021 10:01:47", "content": "

in tf tpu i used 756 img size for better result,,it didn't work for me , don't know why,something is wrong with the tf validation strategy i guess

", "rawMarkdown": "in tf tpu i used 756 img size for better result,,it didn't work for me , don't know why,something is wrong with the tf validation strategy i guess", "votes": null }, { "id": "1222947", "postDate": "03/02/2021 10:18:02", "content": "

I haven't tried 756 yet, but my best result was on 600… That's the only model I could use 600 on because else I will exhaust all my TPU hours in 2 days 😅
\nMost of my experiments are on 456 image size…

", "rawMarkdown": "I haven't tried 756 yet, but my best result was on 600... That's the only model I could use 600 on because else I will exhaust all my TPU hours in 2 days 😅\nMost of my experiments are on 456 image size...", "votes": null }, { "id": "1224550", "postDate": "03/02/2021 20:51:09", "content": "

I'm using a single 1080TI for this competition.

\n

I tried to buy a RTX 3090. It would be a great help not only for Kaggle competitions, but also for other works.

\n

Appart from the GPU war… for me it's even more frustrating that it is not possible to buy a new 3090 in my country! 😭
\nIt's a global stock problem… due to cryptocoin miners

", "rawMarkdown": "I'm using a single 1080TI for this competition.\n\nI tried to buy a RTX 3090. It would be a great help not only for Kaggle competitions, but also for other works.\n\nAppart from the GPU war... for me it's even more frustrating that it is not possible to buy a new 3090 in my country! 😭\nIt's a global stock problem... due to cryptocoin miners", "votes": null }, { "id": "1225487", "postDate": "03/03/2021 16:29:40", "content": "

RTX 3060 has protection against crypto miners (reduction of hash rate). https://blogs.nvidia.com/blog/2021/02/18/geforce-cmp/

", "rawMarkdown": "RTX 3060 has protection against crypto miners (reduction of hash rate). https://blogs.nvidia.com/blog/2021/02/18/geforce-cmp/", "votes": null }, { "id": "1225526", "postDate": "03/03/2021 17:00:45", "content": "

I cannot purchase a 3090 neither, or at a double price.

", "rawMarkdown": "I cannot purchase a 3090 neither, or at a double price.", "votes": null }, { "id": "1225572", "postDate": "03/03/2021 18:01:15", "content": "

This protection is something new for the next RTX models.

\n

Now, I'm waiting for the time when the dust settles

", "rawMarkdown": "This protection is something new for the next RTX models.\n\nNow, I'm waiting for the time when the dust settles", "votes": null }, { "id": "1226129", "postDate": "03/04/2021 09:10:18", "content": "

For image size 384, could reach CV 0.958 and have not submit to LB

", "rawMarkdown": "For image size 384, could reach CV 0.958 and have not submit to LB", "votes": null }, { "id": "1226740", "postDate": "03/04/2021 19:48:02", "content": "

Perhaps the organizers of some competitions can use GPU as prizes instead of money.
\nAfter all, many people can't buy GPU.

", "rawMarkdown": "Perhaps the organizers of some competitions can use GPU as prizes instead of money. \nAfter all, many people can't buy GPU.", "votes": null }, { "id": "1226746", "postDate": "03/04/2021 19:57:01", "content": "

omfg that would be so much better :D
\nimpossible to buy a 3090…

", "rawMarkdown": "omfg that would be so much better :D \nimpossible to buy a 3090...", "votes": null }, { "id": "1226777", "postDate": "03/04/2021 20:55:24", "content": "

That would be quite the trophy

", "rawMarkdown": "That would be quite the trophy", "votes": null }, { "id": "1226887", "postDate": "03/05/2021 01:31:05", "content": "

\"Perhaps the organizers of some competitions can use GPU as prizes instead of money.
\nAfter all, many people can't buy GPU.

\n

join the cvpr 2012/Nvidia AI city challenge.

\n

the prize is rtx 3090

", "rawMarkdown": "\"Perhaps the organizers of some competitions can use GPU as prizes instead of money.\nAfter all, many people can't buy GPU.\n\njoin the cvpr 2012/Nvidia AI city challenge.\n\nthe prize is rtx 3090", "votes": null }, { "id": "1227780", "postDate": "03/05/2021 21:06:38", "content": "

maybe they could give more GPU resources to the top N teams on the leaderboard, or to people who achieve some goals during the competition

\n

… but only for people with an sworn statement saying that they don't already own an RTX 3090!

", "rawMarkdown": "maybe they could give more GPU resources to the top N teams on the leaderboard, or to people who achieve some goals during the competition\n\n... but only for people with an sworn statement saying that they don't already own an RTX 3090!", "votes": null } ], "comments": [ { "id": 1222526, "author_name": "ryches", "author_url": "", "post_date": "03/01/2021 21:33:46", "content": "

Yeah, more is always better. I saw some people mention 1024x1024 images and that is pretty much completely out of the question for me and I would consider myself reasonably well equipped.

", "votes": null, "replies": [ { "id": 1222539, "author_name": "serigne", "author_url": "", "post_date": "03/01/2021 21:56:56", "content": "

The first and unique time I could train with 1024x1024 was using TFRec and Kaggle TPU v3 with Bfloat16

\n

But here I use Pytorch and V100 16 Go

", "votes": null, "replies": [] } ] }, { "id": 1222589, "author_name": "mobassir", "author_url": "", "post_date": "03/01/2021 23:45:35", "content": "

That's why kagglers with less compute power focuses on ensemble most of the cases.
\nI dislike gpu war :(
\nI thought as we have gn,frn,ws these days
\nIt will be easy to compete with kagglers having good gpu
\nBut the image size becomes the bottleneck here again
\nI tried to train resnet34 for long time on little bit bigger image size but it Doesn't work.
\n\"Bigger is better\" :(
\nI Forcefully use tensorflow sometimes even though i am pytorch lover
\nWhy i use tensorflow?
\nV3-8 tpu or tpu pod access
\nThat's crazy fast
\nBut i am not very comfortable with tensorflow
\nI wish torch xla had similar power like tf tpu :(

", "votes": null, "replies": [ { "id": 1222861, "author_name": "serigne", "author_url": "", "post_date": "03/02/2021 08:18:07", "content": "

The images size really matters here.
\nAnd my current approach is power greedy. But unfortunately I can't go further than 416x416. And I am already using Gradient Accumulation and all sort of optimizations.

", "votes": null, "replies": [] }, { "id": 1222871, "author_name": "mobassir", "author_url": "", "post_date": "03/02/2021 08:28:19", "content": "

agree,today from my one experiment i just realized that more than 512 img size is better, i don't know if you can really get good result with 416 img size,it will be difficult to get 0.955 with this image size but probably still possible with deeper networks

", "votes": null, "replies": [] }, { "id": 1226129, "author_name": "steamedsheep", "author_url": "", "post_date": "03/04/2021 09:10:18", "content": "

For image size 384, could reach CV 0.958 and have not submit to LB

", "votes": null, "replies": [] } ] }, { "id": 1222897, "author_name": "manabendrarout", "author_url": "", "post_date": "03/02/2021 09:11:24", "content": "

I feel for you brother… I an also having a hard time in this competition due to lack of processing power… This is the first time I am feeling this helpless… I almost drain all of my GPU and TPU hours with 1-2 days remaining and them I have to wait for Saturday 😅
\nBut it is what it is… I am just happy for the learning opportunity and discovering what it could have been with bigger image size…

", "votes": null, "replies": [ { "id": 1222930, "author_name": "mobassir", "author_url": "", "post_date": "03/02/2021 10:01:47", "content": "

in tf tpu i used 756 img size for better result,,it didn't work for me , don't know why,something is wrong with the tf validation strategy i guess

", "votes": null, "replies": [] }, { "id": 1222947, "author_name": "manabendrarout", "author_url": "", "post_date": "03/02/2021 10:18:02", "content": "

I haven't tried 756 yet, but my best result was on 600… That's the only model I could use 600 on because else I will exhaust all my TPU hours in 2 days 😅
\nMost of my experiments are on 456 image size…

", "votes": null, "replies": [] } ] }, { "id": 1224550, "author_name": "virilo", "author_url": "", "post_date": "03/02/2021 20:51:09", "content": "

I'm using a single 1080TI for this competition.

\n

I tried to buy a RTX 3090. It would be a great help not only for Kaggle competitions, but also for other works.

\n

Appart from the GPU war… for me it's even more frustrating that it is not possible to buy a new 3090 in my country! 😭
\nIt's a global stock problem… due to cryptocoin miners

", "votes": null, "replies": [ { "id": 1225487, "author_name": "atamazian", "author_url": "", "post_date": "03/03/2021 16:29:40", "content": "

RTX 3060 has protection against crypto miners (reduction of hash rate). https://blogs.nvidia.com/blog/2021/02/18/geforce-cmp/

", "votes": null, "replies": [ { "id": 1225572, "author_name": "virilo", "author_url": "", "post_date": "03/03/2021 18:01:15", "content": "

This protection is something new for the next RTX models.

\n

Now, I'm waiting for the time when the dust settles

", "votes": null, "replies": [] } ] }, { "id": 1225526, "author_name": "steamedsheep", "author_url": "", "post_date": "03/03/2021 17:00:45", "content": "

I cannot purchase a 3090 neither, or at a double price.

", "votes": null, "replies": [] } ] }, { "id": 1226740, "author_name": "h053473666", "author_url": "", "post_date": "03/04/2021 19:48:02", "content": "

Perhaps the organizers of some competitions can use GPU as prizes instead of money.
\nAfter all, many people can't buy GPU.

", "votes": null, "replies": [ { "id": 1226746, "author_name": "philippsinger", "author_url": "", "post_date": "03/04/2021 19:57:01", "content": "

omfg that would be so much better :D
\nimpossible to buy a 3090…

", "votes": null, "replies": [] }, { "id": 1226777, "author_name": "ryches", "author_url": "", "post_date": "03/04/2021 20:55:24", "content": "

That would be quite the trophy

", "votes": null, "replies": [] }, { "id": 1226887, "author_name": "hengck23", "author_url": "", "post_date": "03/05/2021 01:31:05", "content": "

\"Perhaps the organizers of some competitions can use GPU as prizes instead of money.
\nAfter all, many people can't buy GPU.

\n

join the cvpr 2012/Nvidia AI city challenge.

\n

the prize is rtx 3090

", "votes": null, "replies": [] } ] }, { "id": 1227780, "author_name": "virilo", "author_url": "", "post_date": "03/05/2021 21:06:38", "content": "

maybe they could give more GPU resources to the top N teams on the leaderboard, or to people who achieve some goals during the competition

\n

… but only for people with an sworn statement saying that they don't already own an RTX 3090!

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1222480": "The idea I am trying right now is showing promising results but I am forced to use relatively low resolutions given I am limited by compute power . \nI am 100% sure I would have much better results with higher resolutions and train faster with better compute power. It's a bit frustrating^^", "1222526": "Yeah, more is always better. I saw some people mention 1024x1024 images and that is pretty much completely out of the question for me and I would consider myself reasonably well equipped.", "1222539": "The first and unique time I could train with 1024x1024 was using TFRec and Kaggle TPU v3 with Bfloat16\n\nBut here I use Pytorch and V100 16 Go", "1222589": "That's why kagglers with less compute power focuses on ensemble most of the cases.\nI dislike gpu war :(\nI thought as we have gn,frn,ws these days\nIt will be easy to compete with kagglers having good gpu\nBut the image size becomes the bottleneck here again\nI tried to train resnet34 for long time on little bit bigger image size but it Doesn't work.\n\"Bigger is better\" :(\nI Forcefully use tensorflow sometimes even though i am pytorch lover\nWhy i use tensorflow? \nV3-8 tpu or tpu pod access\nThat's crazy fast\nBut i am not very comfortable with tensorflow \nI wish torch xla had similar power like tf tpu :(", "1222861": "The images size really matters here.\nAnd my current approach is power greedy. But unfortunately I can't go further than 416x416. And I am already using Gradient Accumulation and all sort of optimizations.", "1222871": "agree,today from my one experiment i just realized that more than 512 img size is better, i don't know if you can really get good result with 416 img size,it will be difficult to get 0.955 with this image size but probably still possible with deeper networks", "1222897": "I feel for you brother... I an also having a hard time in this competition due to lack of processing power... This is the first time I am feeling this helpless... I almost drain all of my GPU and TPU hours with 1-2 days remaining and them I have to wait for Saturday 😅\nBut it is what it is... I am just happy for the learning opportunity and discovering what it could have been with bigger image size...", "1222930": "in tf tpu i used 756 img size for better result,,it didn't work for me , don't know why,something is wrong with the tf validation strategy i guess", "1222947": "I haven't tried 756 yet, but my best result was on 600... That's the only model I could use 600 on because else I will exhaust all my TPU hours in 2 days 😅\nMost of my experiments are on 456 image size...", "1224550": "I'm using a single 1080TI for this competition.\n\nI tried to buy a RTX 3090. It would be a great help not only for Kaggle competitions, but also for other works.\n\nAppart from the GPU war... for me it's even more frustrating that it is not possible to buy a new 3090 in my country! 😭\nIt's a global stock problem... due to cryptocoin miners", "1225487": "RTX 3060 has protection against crypto miners (reduction of hash rate). https://blogs.nvidia.com/blog/2021/02/18/geforce-cmp/", "1225526": "I cannot purchase a 3090 neither, or at a double price.", "1225572": "This protection is something new for the next RTX models.\n\nNow, I'm waiting for the time when the dust settles", "1226129": "For image size 384, could reach CV 0.958 and have not submit to LB", "1226740": "Perhaps the organizers of some competitions can use GPU as prizes instead of money. \nAfter all, many people can't buy GPU.", "1226746": "omfg that would be so much better :D \nimpossible to buy a 3090...", "1226777": "That would be quite the trophy", "1226887": "\"Perhaps the organizers of some competitions can use GPU as prizes instead of money.\nAfter all, many people can't buy GPU.\n\njoin the cvpr 2012/Nvidia AI city challenge.\n\nthe prize is rtx 3090", "1227780": "maybe they could give more GPU resources to the top N teams on the leaderboard, or to people who achieve some goals during the competition\n\n... but only for people with an sworn statement saying that they don't already own an RTX 3090!" }, "source": "meta" }