{ "id": 254775, "title": "Time it takes to train on GPU ", "url": "/competitions/g2net-gravitational-wave-detection/discussion/254775", "author_name": "Sayantan Sadhu", "post_date": "2021-07-23T15:53:01.318000", "votes": 11, "comment_count": 49, "views": 0, "content": "

Hey everyone,
\nI am learning to participate in featured competitions by working on this competitions but I am facing a problem. I am trying to train a 'tf_efficientnet_b7_ns' from this repository 'https://github.com/rwightman/pytorch-image-models' but it is taking a very long time to even go through one epoch using the GPU of Kaggle.
\nI would like to know how much time it took you all if you trained it on Kaggle ?
\nAnd what is the best way ( better if I don't have to buy a GPU 😩) to train these models ?

\n

Thank you

", "messages": [ { "id": 1407472, "postDate": "2021-08-01T22:42:10.490Z", "content": "

My current best model takes ~10 min per epoch on 2x2080Ti (I didn't try to run it on kaggle, but if IO is not too bad the speed should be ~3 times slower because P100 do not provide boost for half precision in comparison to 2080Ti/V100). But I so far considered only relatively small models and low res. The important thing here is looking more into the data and getting the important things done rather than just blindly taking the largest possible model (like many people refer to efficientnet_b7) and the largest resolution, and ending up spending a month to get the model just trained (while getting some result only slightly better than public benchmarks).

\n

I understand that some ppl may have 10s V100 24x7 and could do such wasteful things, but the competitions should be more about finding some interesting things about the data, new approach, creative tricks… While getting anything good using just kaggle resources is very difficult nowadays given the 30 hours week limit, having a computer with just 1-2 2080Ti is quite enough for most of kaggle competitions if one is running experiments wisely.

", "rawMarkdown": "My current best model takes ~10 min per epoch on 2x2080Ti (I didn't try to run it on kaggle, but if IO is not too bad the speed should be ~3 times slower because P100 do not provide boost for half precision in comparison to 2080Ti/V100). But I so far considered only relatively small models and low res. The important thing here is looking more into the data and getting the important things done rather than just blindly taking the largest possible model (like many people refer to efficientnet_b7) and the largest resolution, and ending up spending a month to get the model just trained (while getting some result only slightly better than public benchmarks).\n\nI understand that some ppl may have 10s V100 24x7 and could do such wasteful things, but the competitions should be more about finding some interesting things about the data, new approach, creative tricks... While getting anything good using just kaggle resources is very difficult nowadays given the 30 hours week limit, having a computer with just 1-2 2080Ti is quite enough for most of kaggle competitions if one is running experiments wisely. ", "votes": 15, "replies": [ { "id": 1408010, "postDate": "2021-08-02T08:52:15.657Z", "content": "
\n

I understand that some ppl may have 10s V100 24x7

\n
\n

I wish it was me.

", "rawMarkdown": "> I understand that some ppl may have 10s V100 24x7 \n\nI wish it was me.", "votes": 5 }, { "id": 1408019, "postDate": "2021-08-02T08:58:15.080Z", "content": "

Well I think you are one of those who have 10s A100 24*7

", "rawMarkdown": "Well I think you are one of those who have 10s A100 24*7", "votes": 1 }, { "id": 1408095, "postDate": "2021-08-02T09:53:59.487Z", "content": "

I'm looking forward to someone breaking the legendary record achieved by PFN team at Open Images 2018 - 512 V100s :)
\nhttps://www.kaggle.com/c/google-ai-open-images-object-detection-track/discussion/64986

", "rawMarkdown": "I'm looking forward to someone breaking the legendary record achieved by [PFN](https://www.preferred.jp/en/) team at Open Images 2018 - 512 V100s :)\nhttps://www.kaggle.com/c/google-ai-open-images-object-detection-track/discussion/64986", "votes": 4 }, { "id": 1408267, "postDate": "2021-08-02T11:53:45.337Z", "content": "
\n

Well I think you are on those who have 10s A100 24*7

\n
\n

I have not access to any A100. And as I wrote, I don't have access to 10s of V100 either.

\n

I do have access to some V100 though, which is more than many Kagglers. But I do know people outside NVIDIA who have used 100 V100 in a recent competition.

", "rawMarkdown": "> Well I think you are on those who have 10s A100 24*7\n\nI have not access to any A100. And as I wrote, I don't have access to 10s of V100 either.\n\nI do have access to some V100 though, which is more than many Kagglers. But I do know people outside NVIDIA who have used 100 V100 in a recent competition.", "votes": 1 }, { "id": 1408270, "postDate": "2021-08-02T11:54:26.143Z", "content": "
\n

I have not access to any A100.

\n
\n

reason is customer demand is too strong for now.

", "rawMarkdown": "> I have not access to any A100. \n\nreason is customer demand is too strong for now.", "votes": 6 }, { "id": 1408377, "postDate": "2021-08-02T12:41:53.330Z", "rawMarkdown": "", "votes": 2, "isDeleted": true }, { "id": 1491772, "postDate": "2021-08-26T16:01:33.340Z", "content": "
\n

100 V100 in a recent competition

\n
\n

Yeah, that's a train engine!
\nWith Kaggle competition credits one can run it for a few hours with preemptible vm, everything is possible for all :) but I guess that team didn't squeezed that kind of limit ;)

", "rawMarkdown": "> 100 V100 in a recent competition\n\nYeah, that's a train engine!\nWith Kaggle competition credits one can run it for a few hours with preemptible vm, everything is possible for all :) but I guess that team didn't squeezed that kind of limit ;)" } ] }, { "id": 1397912, "postDate": "2021-07-23T15:53:01.320Z", "content": "

Hey everyone,
\nI am learning to participate in featured competitions by working on this competitions but I am facing a problem. I am trying to train a 'tf_efficientnet_b7_ns' from this repository 'https://github.com/rwightman/pytorch-image-models' but it is taking a very long time to even go through one epoch using the GPU of Kaggle.
\nI would like to know how much time it took you all if you trained it on Kaggle ?
\nAnd what is the best way ( better if I don't have to buy a GPU 😩) to train these models ?

\n

Thank you

", "rawMarkdown": "Hey everyone, \nI am learning to participate in featured competitions by working on this competitions but I am facing a problem. I am trying to train a 'tf_efficientnet_b7_ns' from this repository 'https://github.com/rwightman/pytorch-image-models' but it is taking a very long time to even go through one epoch using the GPU of Kaggle. \nI would like to know how much time it took you all if you trained it on Kaggle ? \nAnd what is the best way ( better if I don't have to buy a GPU 😩) to train these models ?\n\nThank you ", "votes": 11 }, { "id": 1408136, "postDate": "2021-08-02T10:29:57.193Z", "content": "

for me, 1 epoch is trained for 90 minutes for a size of 256 on a gtx1060 (EfficientNetB1)

", "rawMarkdown": "for me, 1 epoch is trained for 90 minutes for a size of 256 on a gtx1060 (EfficientNetB1)", "votes": 6 }, { "id": 1398930, "postDate": "2021-07-24T16:52:52.060Z", "content": "

Well for this type of competition i suggest you to buy colab pro. Its about 10 Dollars a month (800 rupees) and provides you with P100 and in my experience every alternate day a V100 which with Mixed precision can be 2x faster than the p100. Plus the Cpu and the P100 on colab are better for some reason and I think it uses a SSD which is pretty nice for data loading I guess. The main problem you might face is downloading the dataset. I did some experiments and I found this is the best way to download the train data on Colab Pro VM
\ndata_link = \"https://storage.googleapis.com/kaggle-competitions-data/kaggle-v2/23249/2399555/compressed/train.zip?GoogleAccessId=web-data@kaggle-161607.iam.gserviceaccount.com&Expires=1627361246&Signature=OaAJ6vFNpBdjDdj3r6QkIygelxNiLe8irDUjTYf58RV%2FxVXfdLTBcgQeSOZW21yxviqHuHj9fIF7LXVSCWAD6l36mkImIxL9Dt4gMHajFsmpjC3BMZ9SGeln%2FbVQAlweni%2FL%2FOTRTGH5udBU7ZbkO9pmtmBcUT%2FiKHPxK5hUQyrvS3YtbcyELfSHIRa%2B0hDeiGvSynAONwVXEddPaZDJ6hJ02gdKKlU6IGGBL5p4T4%2FR2khZmlcPCFCYpATIteJoA5FLANt1Y0CELLAeyZQpjMnKw%2FFaK%2BIMq31qoVpjKenhkL1GozWTchwC9Xwc2K7v4WOTvxor31obuQgcwp54fg%3D%3D&response-content-disposition=attachment%3B+filename%3Dtrain.zip\"

\n

!wget \"$data_link\" -O train.zip\n!unzip -qq /content/train.zip -d /content/Train\n!rm /content/train.zip

", "rawMarkdown": "Well for this type of competition i suggest you to buy colab pro. Its about 10 Dollars a month (800 rupees) and provides you with P100 and in my experience every alternate day a V100 which with Mixed precision can be 2x faster than the p100. Plus the Cpu and the P100 on colab are better for some reason and I think it uses a SSD which is pretty nice for data loading I guess. The main problem you might face is downloading the dataset. I did some experiments and I found this is the best way to download the train data on Colab Pro VM \n`data_link = \"https://storage.googleapis.com/kaggle-competitions-data/kaggle-v2/23249/2399555/compressed/train.zip?GoogleAccessId=web-data@kaggle-161607.iam.gserviceaccount.com&Expires=1627361246&Signature=OaAJ6vFNpBdjDdj3r6QkIygelxNiLe8irDUjTYf58RV%2FxVXfdLTBcgQeSOZW21yxviqHuHj9fIF7LXVSCWAD6l36mkImIxL9Dt4gMHajFsmpjC3BMZ9SGeln%2FbVQAlweni%2FL%2FOTRTGH5udBU7ZbkO9pmtmBcUT%2FiKHPxK5hUQyrvS3YtbcyELfSHIRa%2B0hDeiGvSynAONwVXEddPaZDJ6hJ02gdKKlU6IGGBL5p4T4%2FR2khZmlcPCFCYpATIteJoA5FLANt1Y0CELLAeyZQpjMnKw%2FFaK%2BIMq31qoVpjKenhkL1GozWTchwC9Xwc2K7v4WOTvxor31obuQgcwp54fg%3D%3D&response-content-disposition=attachment%3B+filename%3Dtrain.zip\"`\n\n`!wget \"$data_link\" -O train.zip\n!unzip -qq /content/train.zip -d /content/Train\n!rm /content/train.zip`", "votes": 3, "replies": [ { "id": 1399227, "postDate": "2021-07-25T03:53:14.057Z", "content": "

Thank you so much I will check it out surely !!!

", "rawMarkdown": "Thank you so much I will check it out surely !!!" }, { "id": 1400946, "postDate": "2021-07-26T18:21:24.327Z", "content": "

Thanks! How did you reduce the size of the file?

", "rawMarkdown": "Thanks! How did you reduce the size of the file?" }, { "id": 1404546, "postDate": "2021-07-30T03:01:09.867Z", "content": "

I did not reduce the size of the File

", "rawMarkdown": "I did not reduce the size of the File " }, { "id": 1406417, "postDate": "2021-07-31T18:47:41.160Z", "content": "

Thanks, Mithil!

", "rawMarkdown": "Thanks, Mithil!" } ] }, { "id": 1398199, "postDate": "2021-07-23T20:34:33.197Z", "content": "

One epoch for our models takes about 40 minutes on a single V100 GPU. On kaggle kernel it maybe slower, but less than 2x slower.

", "rawMarkdown": "One epoch for our models takes about 40 minutes on a single V100 GPU. On kaggle kernel it maybe slower, but less than 2x slower.", "votes": 4, "replies": [ { "id": 1398339, "postDate": "2021-07-24T04:41:56.120Z", "content": "

Thank you so much, It is taking a longer time for me, probably I am making some mistakes.

", "rawMarkdown": "Thank you so much, It is taking a longer time for me, probably I am making some mistakes.", "votes": 1 }, { "id": 1398560, "postDate": "2021-07-24T09:29:58.630Z", "content": "

I always spend time on making training as fast as possible.

", "rawMarkdown": "I always spend time on making training as fast as possible. ", "votes": 2 }, { "id": 1398828, "postDate": "2021-07-24T14:36:48.280Z", "content": "

I am using Pytorch dataset class and dataloader, Any Advice from your side ?

", "rawMarkdown": "I am using Pytorch dataset class and dataloader, Any Advice from your side ?" }, { "id": 1398884, "postDate": "2021-07-24T15:49:29.473Z", "content": "

Many things:

\n", "rawMarkdown": "Many things:\n- Move computation to GPU (eg use torch.fft, or torch.librosa or nn.audio, ect)\n- Use asynchronous data transfer to GPU\n- Use smaller images/model\n- profile your code to see where you spend time\n- use torch.cuda.amp", "votes": 17 }, { "id": 1399226, "postDate": "2021-07-25T03:51:57.637Z", "content": "

Thank you I will try and use them.

", "rawMarkdown": "Thank you I will try and use them. " }, { "id": 1399243, "postDate": "2021-07-25T04:39:04.490Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 1399255, "postDate": "2021-07-25T05:15:58.283Z", "content": "

It may, you have to try and see what works.

\n

I listed what could speed up your code. Then you have to see what speeds your code yet does not degrade your score.

", "rawMarkdown": "It may, you have to try and see what works.\n\nI listed what could speed up your code. Then you have to see what speeds your code yet does not degrade your score.", "votes": 1 }, { "id": 1399299, "postDate": "2021-07-25T07:00:28.250Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1399983, "postDate": "2021-07-25T22:06:34.057Z", "content": "

@cpmpml Thanks a lot for listing these things that could speed up the code! It's really helpful for me. May I ask how I can achieve asynchronous data transfer to GPU? Is it some package or parameters of Dataloader?

", "rawMarkdown": "@cpmpml Thanks a lot for listing these things that could speed up the code! It's really helpful for me. May I ask how I can achieve asynchronous data transfer to GPU? Is it some package or parameters of Dataloader? " }, { "id": 1400261, "postDate": "2021-07-26T06:45:12.913Z", "content": "

It is an argument you set for to(device) or cuda() calls.

\n

There are many more ways to speedup pytorch code, see the tuning guide: https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html

", "rawMarkdown": "It is an argument you set for to(device) or cuda() calls.\n\nThere are many more ways to speedup pytorch code, see the tuning guide: https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html\n", "votes": 5 }, { "id": 1400282, "postDate": "2021-07-26T07:05:05.270Z", "content": "

@cpmpml Thank you very much for sharing the training time of 1 epoch on your model. May I ask you to share the gpu environment you are using? Are you using 8 v100s??

", "rawMarkdown": "@cpmpml Thank you very much for sharing the training time of 1 epoch on your model. May I ask you to share the gpu environment you are using? Are you using 8 v100s??" }, { "id": 1400304, "postDate": "2021-07-26T07:34:24.333Z", "content": "

4 V100, DGX Station V100.

", "rawMarkdown": "4 V100, DGX Station V100.", "votes": 3 }, { "id": 1400422, "postDate": "2021-07-26T09:18:31.983Z", "content": "

Very thanks :)

", "rawMarkdown": "Very thanks :)", "votes": 1 }, { "id": 1400613, "postDate": "2021-07-26T12:33:55.583Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 1400644, "postDate": "2021-07-26T13:17:19.753Z", "content": "

I mean if you are serious enough then yes . The budget should what the maximum you can afford with 40 percent on a good gpu,25 percent on a cpu and a good power supply is essential for Deep learning.

", "rawMarkdown": "I mean if you are serious enough then yes . The budget should what the maximum you can afford with 40 percent on a good gpu,25 percent on a cpu and a good power supply is essential for Deep learning. " }, { "id": 1400757, "postDate": "2021-07-26T14:59:00.813Z", "content": "

Thank you so much!

", "rawMarkdown": "Thank you so much!" }, { "id": 1400832, "postDate": "2021-07-26T16:12:46.353Z", "content": "

@ayhmrba I bought a custom PC with 2 GTX 1080 Ti wih my first Kaggle prize, it has been a game changer.

\n

In the past there were lots of tabular competitions where having a good PC was enough with XGBoost. Now most competitions are deep learning competitions and having a recent GPU is certainly worth it. Kaggle, Colab are great free offerings, but as soon as you need more GPU time then cloud solutions are becoming more expensive than acquiring a GPU.

\n

( Well, that was true before the GPU shortage. It seems this shortage is being resolved progressively.)

", "rawMarkdown": "@ayhmrba I bought a custom PC with 2 GTX 1080 Ti wih my first Kaggle prize, it has been a game changer.\n\nIn the past there were lots of tabular competitions where having a good PC was enough with XGBoost. Now most competitions are deep learning competitions and having a recent GPU is certainly worth it. Kaggle, Colab are great free offerings, but as soon as you need more GPU time then cloud solutions are becoming more expensive than acquiring a GPU.\n\n( Well, that was true before the GPU shortage. It seems this shortage is being resolved progressively.)", "votes": 4 }, { "id": 1400837, "postDate": "2021-07-26T16:17:52.223Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 1400950, "postDate": "2021-07-26T18:27:01.263Z", "content": "

The only thing I can share now is to try stuff, try as many things you can. You don't need to run training till completion to see if something is promising. One fold, and first few epochs for that fold maybe enough. Try stuff, even what looks crazy.

\n

Also, team, because other people idea may help you a lot.

", "rawMarkdown": "The only thing I can share now is to try stuff, try as many things you can. You don't need to run training till completion to see if something is promising. One fold, and first few epochs for that fold maybe enough. Try stuff, even what looks crazy. \n\nAlso, team, because other people idea may help you a lot.", "votes": 5 }, { "id": 1401006, "postDate": "2021-07-26T19:58:16.750Z", "rawMarkdown": "", "votes": 2, "isDeleted": true }, { "id": 1408599, "postDate": "2021-08-02T14:11:57.140Z", "content": "

On Kaggle, it's also useful to precompute some transformations and save the result to a dataset/notebook output if you are going to reuse it over and over again (after experimentation stage).

", "rawMarkdown": "On Kaggle, it's also useful to precompute some transformations and save the result to a dataset/notebook output if you are going to reuse it over and over again (after experimentation stage). " }, { "id": 1408632, "postDate": "2021-08-02T14:28:36.260Z", "content": "

Use precomputed things at kaggle may not work well: the main issue of kaggle kernel is insufficient CPU compute( So if one is trying to load images at kaggle, it may be slower in comparison with nnAudio done at GPU.
\nAlso the size of data vs accuracy may be an issue: it may be good to use data without any artifacts resulted by conversion of spectrogram to images (uint8), while saving spectrogram without such conversion may result in several times larger dataset than the initial data(

", "rawMarkdown": "Use precomputed things at kaggle may not work well: the main issue of kaggle kernel is insufficient CPU compute( So if one is trying to load images at kaggle, it may be slower in comparison with nnAudio done at GPU. \nAlso the size of data vs accuracy may be an issue: it may be good to use data without any artifacts resulted by conversion of spectrogram to images (uint8), while saving spectrogram without such conversion may result in several times larger dataset than the initial data(", "votes": 4 }, { "id": 1451868, "postDate": "2021-08-05T13:01:52.877Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 1406173, "postDate": "2021-07-31T15:02:11.307Z", "content": "

You can use TPU instead of GPU - it should be faster if you'll use TPU's 8 cores.

", "rawMarkdown": "You can use TPU instead of GPU - it should be faster if you'll use TPU's 8 cores.", "votes": 1 }, { "id": 1404186, "postDate": "2021-07-29T16:16:48.917Z", "content": "

On using Tensorflow with appropriate TPU configurations, it takes about an hour per epoch using 256x256 as the image size to train a EfficientNetB7

", "rawMarkdown": "On using Tensorflow with appropriate TPU configurations, it takes about an hour per epoch using 256x256 as the image size to train a `EfficientNetB7`", "votes": 1, "replies": [ { "id": 1405133, "postDate": "2021-07-30T14:10:10.800Z", "content": "

Thanks for the information !!!

", "rawMarkdown": "Thanks for the information !!!" } ] }, { "id": 1561000, "postDate": "2021-10-27T09:03:49.943Z", "content": "

Hey All,

\n

Thank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.

\n

Thank you all,

\n

Regards,
\nChris

", "rawMarkdown": "Hey All,\n\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.\n\nThank you all,\n\nRegards,\nChris" }, { "id": 1559988, "postDate": "2021-10-27T08:53:15.057Z", "content": "

Hey All,

\n

Thank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.

\n

Thank you all,

\n

Regards,
\nChris

", "rawMarkdown": "Hey All,\n\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.\n\nThank you all,\n\nRegards,\nChris" }, { "id": 1408589, "postDate": "2021-08-02T14:07:31.700Z", "content": "

For efficientnet b4 it takes about 45-55 minutes per epoch. Efficientnet b7 is far far far more parameters than b4 as b4 is the highest in the better-axis. (there's a graph showing accuracy vs tpu times, and then there's a 3rd axis called the \"better\" axis). So you can see how much more time it will take for B7 (perhaps 2-3 hours per epoch?)

", "rawMarkdown": "For efficientnet b4 it takes about 45-55 minutes per epoch. Efficientnet b7 is far far far more parameters than b4 as b4 is the highest in the better-axis. (there's a graph showing accuracy vs tpu times, and then there's a 3rd axis called the \"better\" axis). So you can see how much more time it will take for B7 (perhaps 2-3 hours per epoch?)" }, { "id": 1408453, "postDate": "2021-08-02T13:12:26.633Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1398124, "postDate": "2021-07-23T18:41:50.963Z", "rawMarkdown": "", "votes": 2, "isDeleted": true, "replies": [ { "id": 1398150, "postDate": "2021-07-23T19:23:42.207Z", "content": "

Thank you so much !!!
\nJust a question would you mind telling how long it took you to train ? Then it would help me to understand what I am doing is correct or not.
\nThanks again.

", "rawMarkdown": "Thank you so much !!! \nJust a question would you mind telling how long it took you to train ? Then it would help me to understand what I am doing is correct or not. \nThanks again. " }, { "id": 1398177, "postDate": "2021-07-23T19:49:49.947Z", "rawMarkdown": "", "votes": 2, "isDeleted": true }, { "id": 1398340, "postDate": "2021-07-24T04:42:50.160Z", "content": "

Thanks the information would help me.

", "rawMarkdown": "Thanks the information would help me." } ] } ], "comments": [ { "id": 1407472, "author_name": "Iafoss", "author_url": "", "post_date": "2021-08-01T22:42:10.490000", "content": "

My current best model takes ~10 min per epoch on 2x2080Ti (I didn't try to run it on kaggle, but if IO is not too bad the speed should be ~3 times slower because P100 do not provide boost for half precision in comparison to 2080Ti/V100). But I so far considered only relatively small models and low res. The important thing here is looking more into the data and getting the important things done rather than just blindly taking the largest possible model (like many people refer to efficientnet_b7) and the largest resolution, and ending up spending a month to get the model just trained (while getting some result only slightly better than public benchmarks).

\n

I understand that some ppl may have 10s V100 24x7 and could do such wasteful things, but the competitions should be more about finding some interesting things about the data, new approach, creative tricks… While getting anything good using just kaggle resources is very difficult nowadays given the 30 hours week limit, having a computer with just 1-2 2080Ti is quite enough for most of kaggle competitions if one is running experiments wisely.

", "votes": 15, "replies": [ { "id": 1408010, "author_name": "CPMP", "author_url": "", "post_date": "2021-08-02T08:52:15.657000", "content": "
\n

I understand that some ppl may have 10s V100 24x7

\n
\n

I wish it was me.

", "votes": 5, "replies": [] }, { "id": 1408019, "author_name": "Mithil Salunkhe", "author_url": "", "post_date": "2021-08-02T08:58:15.080000", "content": "

Well I think you are one of those who have 10s A100 24*7

", "votes": 1, "replies": [] }, { "id": 1408095, "author_name": "RabotniKuma", "author_url": "", "post_date": "2021-08-02T09:53:59.487000", "content": "

I'm looking forward to someone breaking the legendary record achieved by PFN team at Open Images 2018 - 512 V100s :)
\nhttps://www.kaggle.com/c/google-ai-open-images-object-detection-track/discussion/64986

", "votes": 4, "replies": [] }, { "id": 1408267, "author_name": "CPMP", "author_url": "", "post_date": "2021-08-02T11:53:45.337000", "content": "
\n

Well I think you are on those who have 10s A100 24*7

\n
\n

I have not access to any A100. And as I wrote, I don't have access to 10s of V100 either.

\n

I do have access to some V100 though, which is more than many Kagglers. But I do know people outside NVIDIA who have used 100 V100 in a recent competition.

", "votes": 1, "replies": [] }, { "id": 1408270, "author_name": "CPMP", "author_url": "", "post_date": "2021-08-02T11:54:26.143000", "content": "
\n

I have not access to any A100.

\n
\n

reason is customer demand is too strong for now.

", "votes": 6, "replies": [] }, { "id": 1408377, "author_name": "", "author_url": "", "post_date": "2021-08-02T12:41:53.330000", "content": "", "votes": 2, "replies": [] }, { "id": 1491772, "author_name": "Kirderf", "author_url": "", "post_date": "2021-08-26T16:01:33.340000", "content": "
\n

100 V100 in a recent competition

\n
\n

Yeah, that's a train engine!
\nWith Kaggle competition credits one can run it for a few hours with preemptible vm, everything is possible for all :) but I guess that team didn't squeezed that kind of limit ;)

", "votes": 0, "replies": [] } ] }, { "id": 1408136, "author_name": "Kramarenko Vladislav", "author_url": "", "post_date": "2021-08-02T10:29:57.193000", "content": "

for me, 1 epoch is trained for 90 minutes for a size of 256 on a gtx1060 (EfficientNetB1)

", "votes": 6, "replies": [] }, { "id": 1398930, "author_name": "Mithil Salunkhe", "author_url": "", "post_date": "2021-07-24T16:52:52.060000", "content": "

Well for this type of competition i suggest you to buy colab pro. Its about 10 Dollars a month (800 rupees) and provides you with P100 and in my experience every alternate day a V100 which with Mixed precision can be 2x faster than the p100. Plus the Cpu and the P100 on colab are better for some reason and I think it uses a SSD which is pretty nice for data loading I guess. The main problem you might face is downloading the dataset. I did some experiments and I found this is the best way to download the train data on Colab Pro VM
\ndata_link = \"https://storage.googleapis.com/kaggle-competitions-data/kaggle-v2/23249/2399555/compressed/train.zip?GoogleAccessId=web-data@kaggle-161607.iam.gserviceaccount.com&Expires=1627361246&Signature=OaAJ6vFNpBdjDdj3r6QkIygelxNiLe8irDUjTYf58RV%2FxVXfdLTBcgQeSOZW21yxviqHuHj9fIF7LXVSCWAD6l36mkImIxL9Dt4gMHajFsmpjC3BMZ9SGeln%2FbVQAlweni%2FL%2FOTRTGH5udBU7ZbkO9pmtmBcUT%2FiKHPxK5hUQyrvS3YtbcyELfSHIRa%2B0hDeiGvSynAONwVXEddPaZDJ6hJ02gdKKlU6IGGBL5p4T4%2FR2khZmlcPCFCYpATIteJoA5FLANt1Y0CELLAeyZQpjMnKw%2FFaK%2BIMq31qoVpjKenhkL1GozWTchwC9Xwc2K7v4WOTvxor31obuQgcwp54fg%3D%3D&response-content-disposition=attachment%3B+filename%3Dtrain.zip\"

\n

!wget \"$data_link\" -O train.zip\n!unzip -qq /content/train.zip -d /content/Train\n!rm /content/train.zip

", "votes": 3, "replies": [ { "id": 1399227, "author_name": "Sayantan Sadhu", "author_url": "", "post_date": "2021-07-25T03:53:14.057000", "content": "

Thank you so much I will check it out surely !!!

", "votes": 0, "replies": [] }, { "id": 1400946, "author_name": "Richard Xing", "author_url": "", "post_date": "2021-07-26T18:21:24.327000", "content": "

Thanks! How did you reduce the size of the file?

", "votes": 0, "replies": [] }, { "id": 1404546, "author_name": "Mithil Salunkhe", "author_url": "", "post_date": "2021-07-30T03:01:09.867000", "content": "

I did not reduce the size of the File

", "votes": 0, "replies": [] }, { "id": 1406417, "author_name": "Richard Xing", "author_url": "", "post_date": "2021-07-31T18:47:41.160000", "content": "

Thanks, Mithil!

", "votes": 0, "replies": [] } ] }, { "id": 1398199, "author_name": "CPMP", "author_url": "", "post_date": "2021-07-23T20:34:33.197000", "content": "

One epoch for our models takes about 40 minutes on a single V100 GPU. On kaggle kernel it maybe slower, but less than 2x slower.

", "votes": 4, "replies": [ { "id": 1398339, "author_name": "Sayantan Sadhu", "author_url": "", "post_date": "2021-07-24T04:41:56.120000", "content": "

Thank you so much, It is taking a longer time for me, probably I am making some mistakes.

", "votes": 1, "replies": [] }, { "id": 1398560, "author_name": "CPMP", "author_url": "", "post_date": "2021-07-24T09:29:58.630000", "content": "

I always spend time on making training as fast as possible.

", "votes": 2, "replies": [] }, { "id": 1398828, "author_name": "Sayantan Sadhu", "author_url": "", "post_date": "2021-07-24T14:36:48.280000", "content": "

I am using Pytorch dataset class and dataloader, Any Advice from your side ?

", "votes": 0, "replies": [] }, { "id": 1398884, "author_name": "CPMP", "author_url": "", "post_date": "2021-07-24T15:49:29.473000", "content": "

Many things:

\n", "votes": 17, "replies": [] }, { "id": 1399226, "author_name": "Sayantan Sadhu", "author_url": "", "post_date": "2021-07-25T03:51:57.637000", "content": "

Thank you I will try and use them.

", "votes": 0, "replies": [] }, { "id": 1399243, "author_name": "", "author_url": "", "post_date": "2021-07-25T04:39:04.490000", "content": "", "votes": 1, "replies": [] }, { "id": 1399255, "author_name": "CPMP", "author_url": "", "post_date": "2021-07-25T05:15:58.283000", "content": "

It may, you have to try and see what works.

\n

I listed what could speed up your code. Then you have to see what speeds your code yet does not degrade your score.

", "votes": 1, "replies": [] }, { "id": 1399299, "author_name": "", "author_url": "", "post_date": "2021-07-25T07:00:28.250000", "content": "", "votes": 0, "replies": [] }, { "id": 1399983, "author_name": "Richard Xing", "author_url": "", "post_date": "2021-07-25T22:06:34.057000", "content": "

@cpmpml Thanks a lot for listing these things that could speed up the code! It's really helpful for me. May I ask how I can achieve asynchronous data transfer to GPU? Is it some package or parameters of Dataloader?

", "votes": 0, "replies": [] }, { "id": 1400261, "author_name": "CPMP", "author_url": "", "post_date": "2021-07-26T06:45:12.913000", "content": "

It is an argument you set for to(device) or cuda() calls.

\n

There are many more ways to speedup pytorch code, see the tuning guide: https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html

", "votes": 5, "replies": [] }, { "id": 1400282, "author_name": "Jun Kyu Jang", "author_url": "", "post_date": "2021-07-26T07:05:05.270000", "content": "

@cpmpml Thank you very much for sharing the training time of 1 epoch on your model. May I ask you to share the gpu environment you are using? Are you using 8 v100s??

", "votes": 0, "replies": [] }, { "id": 1400304, "author_name": "CPMP", "author_url": "", "post_date": "2021-07-26T07:34:24.333000", "content": "

4 V100, DGX Station V100.

", "votes": 3, "replies": [] }, { "id": 1400422, "author_name": "Jun Kyu Jang", "author_url": "", "post_date": "2021-07-26T09:18:31.983000", "content": "

Very thanks :)

", "votes": 1, "replies": [] }, { "id": 1400613, "author_name": "", "author_url": "", "post_date": "2021-07-26T12:33:55.583000", "content": "", "votes": 1, "replies": [] }, { "id": 1400644, "author_name": "Mithil Salunkhe", "author_url": "", "post_date": "2021-07-26T13:17:19.753000", "content": "

I mean if you are serious enough then yes . The budget should what the maximum you can afford with 40 percent on a good gpu,25 percent on a cpu and a good power supply is essential for Deep learning.

", "votes": 0, "replies": [] }, { "id": 1400757, "author_name": "Richard Xing", "author_url": "", "post_date": "2021-07-26T14:59:00.813000", "content": "

Thank you so much!

", "votes": 0, "replies": [] }, { "id": 1400832, "author_name": "CPMP", "author_url": "", "post_date": "2021-07-26T16:12:46.353000", "content": "

@ayhmrba I bought a custom PC with 2 GTX 1080 Ti wih my first Kaggle prize, it has been a game changer.

\n

In the past there were lots of tabular competitions where having a good PC was enough with XGBoost. Now most competitions are deep learning competitions and having a recent GPU is certainly worth it. Kaggle, Colab are great free offerings, but as soon as you need more GPU time then cloud solutions are becoming more expensive than acquiring a GPU.

\n

( Well, that was true before the GPU shortage. It seems this shortage is being resolved progressively.)

", "votes": 4, "replies": [] }, { "id": 1400837, "author_name": "", "author_url": "", "post_date": "2021-07-26T16:17:52.223000", "content": "", "votes": 1, "replies": [] }, { "id": 1400950, "author_name": "CPMP", "author_url": "", "post_date": "2021-07-26T18:27:01.263000", "content": "

The only thing I can share now is to try stuff, try as many things you can. You don't need to run training till completion to see if something is promising. One fold, and first few epochs for that fold maybe enough. Try stuff, even what looks crazy.

\n

Also, team, because other people idea may help you a lot.

", "votes": 5, "replies": [] }, { "id": 1401006, "author_name": "", "author_url": "", "post_date": "2021-07-26T19:58:16.750000", "content": "", "votes": 2, "replies": [] }, { "id": 1408599, "author_name": "Wabinab Chow", "author_url": "", "post_date": "2021-08-02T14:11:57.140000", "content": "

On Kaggle, it's also useful to precompute some transformations and save the result to a dataset/notebook output if you are going to reuse it over and over again (after experimentation stage).

", "votes": 0, "replies": [] }, { "id": 1408632, "author_name": "Iafoss", "author_url": "", "post_date": "2021-08-02T14:28:36.260000", "content": "

Use precomputed things at kaggle may not work well: the main issue of kaggle kernel is insufficient CPU compute( So if one is trying to load images at kaggle, it may be slower in comparison with nnAudio done at GPU.
\nAlso the size of data vs accuracy may be an issue: it may be good to use data without any artifacts resulted by conversion of spectrogram to images (uint8), while saving spectrogram without such conversion may result in several times larger dataset than the initial data(

", "votes": 4, "replies": [] }, { "id": 1451868, "author_name": "", "author_url": "", "post_date": "2021-08-05T13:01:52.877000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1406173, "author_name": "Araik Tamazian", "author_url": "", "post_date": "2021-07-31T15:02:11.307000", "content": "

You can use TPU instead of GPU - it should be faster if you'll use TPU's 8 cores.

", "votes": 1, "replies": [] }, { "id": 1404186, "author_name": "Saurav Maheshkar ☕️", "author_url": "", "post_date": "2021-07-29T16:16:48.917000", "content": "

On using Tensorflow with appropriate TPU configurations, it takes about an hour per epoch using 256x256 as the image size to train a EfficientNetB7

", "votes": 1, "replies": [ { "id": 1405133, "author_name": "Sayantan Sadhu", "author_url": "", "post_date": "2021-07-30T14:10:10.800000", "content": "

Thanks for the information !!!

", "votes": 0, "replies": [] } ] }, { "id": 1561000, "author_name": "ChristopherZerafa", "author_url": "", "post_date": "2021-10-27T09:03:49.943000", "content": "

Hey All,

\n

Thank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.

\n

Thank you all,

\n

Regards,
\nChris

", "votes": 0, "replies": [] }, { "id": 1559988, "author_name": "ChristopherZerafa", "author_url": "", "post_date": "2021-10-27T08:53:15.057000", "content": "

Hey All,

\n

Thank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.

\n

Thank you all,

\n

Regards,
\nChris

", "votes": 0, "replies": [] }, { "id": 1408589, "author_name": "Wabinab Chow", "author_url": "", "post_date": "2021-08-02T14:07:31.700000", "content": "

For efficientnet b4 it takes about 45-55 minutes per epoch. Efficientnet b7 is far far far more parameters than b4 as b4 is the highest in the better-axis. (there's a graph showing accuracy vs tpu times, and then there's a 3rd axis called the \"better\" axis). So you can see how much more time it will take for B7 (perhaps 2-3 hours per epoch?)

", "votes": 0, "replies": [] }, { "id": 1408453, "author_name": "", "author_url": "", "post_date": "2021-08-02T13:12:26.633000", "content": "", "votes": 0, "replies": [] }, { "id": 1398124, "author_name": "", "author_url": "", "post_date": "2021-07-23T18:41:50.963000", "content": "", "votes": 2, "replies": [ { "id": 1398150, "author_name": "Sayantan Sadhu", "author_url": "", "post_date": "2021-07-23T19:23:42.207000", "content": "

Thank you so much !!!
\nJust a question would you mind telling how long it took you to train ? Then it would help me to understand what I am doing is correct or not.
\nThanks again.

", "votes": 0, "replies": [] }, { "id": 1398177, "author_name": "", "author_url": "", "post_date": "2021-07-23T19:49:49.947000", "content": "", "votes": 2, "replies": [] }, { "id": 1398340, "author_name": "Sayantan Sadhu", "author_url": "", "post_date": "2021-07-24T04:42:50.160000", "content": "

Thanks the information would help me.

", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1407472": "My current best model takes ~10 min per epoch on 2x2080Ti (I didn't try to run it on kaggle, but if IO is not too bad the speed should be ~3 times slower because P100 do not provide boost for half precision in comparison to 2080Ti/V100). But I so far considered only relatively small models and low res. The important thing here is looking more into the data and getting the important things done rather than just blindly taking the largest possible model (like many people refer to efficientnet_b7) and the largest resolution, and ending up spending a month to get the model just trained (while getting some result only slightly better than public benchmarks).\n\nI understand that some ppl may have 10s V100 24x7 and could do such wasteful things, but the competitions should be more about finding some interesting things about the data, new approach, creative tricks... While getting anything good using just kaggle resources is very difficult nowadays given the 30 hours week limit, having a computer with just 1-2 2080Ti is quite enough for most of kaggle competitions if one is running experiments wisely. ", "1397912": "Hey everyone, \nI am learning to participate in featured competitions by working on this competitions but I am facing a problem. I am trying to train a 'tf_efficientnet_b7_ns' from this repository 'https://github.com/rwightman/pytorch-image-models' but it is taking a very long time to even go through one epoch using the GPU of Kaggle. \nI would like to know how much time it took you all if you trained it on Kaggle ? \nAnd what is the best way ( better if I don't have to buy a GPU 😩) to train these models ?\n\nThank you ", "1408136": "for me, 1 epoch is trained for 90 minutes for a size of 256 on a gtx1060 (EfficientNetB1)", "1398930": "Well for this type of competition i suggest you to buy colab pro. Its about 10 Dollars a month (800 rupees) and provides you with P100 and in my experience every alternate day a V100 which with Mixed precision can be 2x faster than the p100. Plus the Cpu and the P100 on colab are better for some reason and I think it uses a SSD which is pretty nice for data loading I guess. The main problem you might face is downloading the dataset. I did some experiments and I found this is the best way to download the train data on Colab Pro VM \n`data_link = \"https://storage.googleapis.com/kaggle-competitions-data/kaggle-v2/23249/2399555/compressed/train.zip?GoogleAccessId=web-data@kaggle-161607.iam.gserviceaccount.com&Expires=1627361246&Signature=OaAJ6vFNpBdjDdj3r6QkIygelxNiLe8irDUjTYf58RV%2FxVXfdLTBcgQeSOZW21yxviqHuHj9fIF7LXVSCWAD6l36mkImIxL9Dt4gMHajFsmpjC3BMZ9SGeln%2FbVQAlweni%2FL%2FOTRTGH5udBU7ZbkO9pmtmBcUT%2FiKHPxK5hUQyrvS3YtbcyELfSHIRa%2B0hDeiGvSynAONwVXEddPaZDJ6hJ02gdKKlU6IGGBL5p4T4%2FR2khZmlcPCFCYpATIteJoA5FLANt1Y0CELLAeyZQpjMnKw%2FFaK%2BIMq31qoVpjKenhkL1GozWTchwC9Xwc2K7v4WOTvxor31obuQgcwp54fg%3D%3D&response-content-disposition=attachment%3B+filename%3Dtrain.zip\"`\n\n`!wget \"$data_link\" -O train.zip\n!unzip -qq /content/train.zip -d /content/Train\n!rm /content/train.zip`", "1398199": "One epoch for our models takes about 40 minutes on a single V100 GPU. On kaggle kernel it maybe slower, but less than 2x slower.", "1406173": "You can use TPU instead of GPU - it should be faster if you'll use TPU's 8 cores.", "1404186": "On using Tensorflow with appropriate TPU configurations, it takes about an hour per epoch using 256x256 as the image size to train a `EfficientNetB7`", "1561000": "Hey All,\n\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.\n\nThank you all,\n\nRegards,\nChris", "1559988": "Hey All,\n\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.\n\nThank you all,\n\nRegards,\nChris", "1408589": "For efficientnet b4 it takes about 45-55 minutes per epoch. Efficientnet b7 is far far far more parameters than b4 as b4 is the highest in the better-axis. (there's a graph showing accuracy vs tpu times, and then there's a 3rd axis called the \"better\" axis). So you can see how much more time it will take for B7 (perhaps 2-3 hours per epoch?)", "1408453": "", "1398124": "" } }