It's a known issue with efficient nets that it is usually slower on Pytorch. Afaik depthwise convolution are really poorly optimized in Pytorch as can be seen in some tickets both on the efficientnet repo: https://github.com/lukemelas/EfficientNet-PyTorch/issues/159
\nand pytorch: https://github.com/pytorch/pytorch/issues/18631
Thankfully there is a recent paper proposing ResNest https://arxiv.org/pdf/2004.08955.pdf and a pytorch implementation here https://github.com/zhanghang1989/ResNeSt. For example for me a B2 and a Resnest50 train in approx the same time but the resnest50 showed a good CV/LB increase since it's a bigger model at 25M instead of the 7M for the B2.
\nSome numbers on my gtx1080ti:
\nB2 7M 7:30 per epoch
\nResnest50 25M 6:45 per epoch
\nB4 19M 20:00 per epoch
\nResnest100 46M 14:00 per epoch
All this is of course slower than TF + TPU :)
", "messages": [ { "id": 962929, "postDate": "2020-08-08T14:54:01.543Z", "content": "It's a known issue with efficient nets that it is usually slower on Pytorch. Afaik depthwise convolution are really poorly optimized in Pytorch as can be seen in some tickets both on the efficientnet repo: https://github.com/lukemelas/EfficientNet-PyTorch/issues/159
\nand pytorch: https://github.com/pytorch/pytorch/issues/18631
Thankfully there is a recent paper proposing ResNest https://arxiv.org/pdf/2004.08955.pdf and a pytorch implementation here https://github.com/zhanghang1989/ResNeSt. For example for me a B2 and a Resnest50 train in approx the same time but the resnest50 showed a good CV/LB increase since it's a bigger model at 25M instead of the 7M for the B2.
\nSome numbers on my gtx1080ti:
\nB2 7M 7:30 per epoch
\nResnest50 25M 6:45 per epoch
\nB4 19M 20:00 per epoch
\nResnest100 46M 14:00 per epoch
All this is of course slower than TF + TPU :)
", "rawMarkdown": "It's a known issue with efficient nets that it is usually slower on Pytorch. Afaik depthwise convolution are really poorly optimized in Pytorch as can be seen in some tickets both on the efficientnet repo: https://github.com/lukemelas/EfficientNet-PyTorch/issues/159\nand pytorch: https://github.com/pytorch/pytorch/issues/18631\n\nThankfully there is a recent paper proposing ResNest https://arxiv.org/pdf/2004.08955.pdf and a pytorch implementation here https://github.com/zhanghang1989/ResNeSt. For example for me a B2 and a Resnest50 train in approx the same time but the resnest50 showed a good CV/LB increase since it's a bigger model at 25M instead of the 7M for the B2.\n\nSome numbers on my gtx1080ti:\nB2 7M 7:30 per epoch\nResnest50 25M 6:45 per epoch\nB4 19M 20:00 per epoch\nResnest100 46M 14:00 per epoch\n\nAll this is of course slower than TF + TPU :)", "votes": 25 }, { "id": 964937, "postDate": "2020-08-10T09:25:41.800Z", "content": "First try isn't good, I guess I have to retune everything.
", "rawMarkdown": "First try isn't good, I guess I have to retune everything.", "votes": 3 }, { "id": 963322, "postDate": "2020-08-08T22:32:50.793Z", "content": "\n\nAll this is of course slower than TF + TPU :)
\n
It's a little bit sad that Pytorch doesn't have a proper way right now to load data directly on TPUs( need to use CPU -> Image data preprocessing is extremely slow at kaggle. So XLA could be really used at kaggle only for NLP, and even their 16 GB RAM limitation makes it difficult to deal with. While TF is not flexible, and it is difficult to add even simple things.
", "rawMarkdown": "> All this is of course slower than TF + TPU :)\n\nIt's a little bit sad that Pytorch doesn't have a proper way right now to load data directly on TPUs( need to use CPU -> Image data preprocessing is extremely slow at kaggle. So XLA could be really used at kaggle only for NLP, and even their 16 GB RAM limitation makes it difficult to deal with. While TF is not flexible, and it is difficult to add even simple things.\n", "votes": 3, "replies": [ { "id": 963334, "postDate": "2020-08-08T23:03:00.687Z", "content": "Yes, exactly my thoughts. If at least there was an equivalent to fast I/O + processing I'd gladly invest modifying my code to use XLA (or use pytorch lightning as I do locally but usually lightning for Kaggle TPU often fails for me).
", "rawMarkdown": "Yes, exactly my thoughts. If at least there was an equivalent to fast I/O + processing I'd gladly invest modifying my code to use XLA (or use pytorch lightning as I do locally but usually lightning for Kaggle TPU often fails for me).", "votes": 1 }, { "id": 963342, "postDate": "2020-08-08T23:33:27.493Z", "content": "I/O + processing is clearly an issue, however Colab Pro has much better config for Pytorch TPU than Kaggle. More CPU cores and +35 GB of RAM.
\nAnd you can run it h24
@serigne is it available in France? So far only available in the US, right?
", "rawMarkdown": "@serigne is it available in France? So far only available in the US, right?" }, { "id": 964031, "postDate": "2020-08-09T14:28:48.907Z", "content": "Yeah it's available. Don't worry about the US zip code, you can put yours.
", "rawMarkdown": "Yeah it's available. Don't worry about the US zip code, you can put yours. ", "votes": 2 }, { "id": 964273, "postDate": "2020-08-09T17:55:04.747Z", "content": "Alright thanks!
", "rawMarkdown": "Alright thanks!" }, { "id": 965539, "postDate": "2020-08-10T17:47:22.077Z", "content": "@serigne Thanks for the colab pro tip. Didn't know that service existed. Price is pretty good too. Using it know for bigger model.
", "rawMarkdown": "@serigne Thanks for the colab pro tip. Didn't know that service existed. Price is pretty good too. Using it know for bigger model." }, { "id": 966057, "postDate": "2020-08-11T05:23:10.470Z", "content": "De rien @arroqc
\nNote that on TPU , you can execute a notebook, close the browser and il will be still running. You can open it hours later elsewhere and find all the training log.
\nThis doesn't work for GPU though. It would disconnect after about 1 hour of no interactive session.
", "rawMarkdown": "De rien @arroqc \n\nNote that on TPU , you can execute a notebook, close the browser and il will be still running. You can open it hours later elsewhere and find all the training log. \n\nThis doesn't work for GPU though. It would disconnect after about 1 hour of no interactive session." }, { "id": 972018, "postDate": "2020-08-16T07:02:18.087Z", "content": "Have you tried NVIDIA-DALI which uses GPU for image augmentations and loads images directly on to GPU which makes data loading much faster as compared to the traditional approach.
", "rawMarkdown": "Have you tried NVIDIA-DALI which uses GPU for image augmentations and loads images directly on to GPU which makes data loading much faster as compared to the traditional approach.", "votes": 5, "replies": [ { "id": 972062, "postDate": "2020-08-16T07:43:10.140Z", "content": "@chandanverma I personally have not tried DALI, but I am going to look into it because that tech sounds interesting. However, my instinct is, I am not sure its a good thing for my current pipeline. GPU memory is a premium for me. I don’t have any GPU memory available to load the next batch and be doing pre-processing while the GPU is doing training. With CPU based augmentations, they are actually super quick,and they use CPU memory. I use pin_memory and a async copies into GPU memory, so my GPU is at 100%. If your GPU is at 100% then its not waiting on you and I can’t imagine GPU augmentations being better. In short, I think CPU + GPU is a powerful combination. For just straight up doing pre-processing, but not online pre-processing, then I think DALI sounds great. Either way, I am going to look into it.
", "rawMarkdown": "@chandanverma I personally have not tried DALI, but I am going to look into it because that tech sounds interesting. However, my instinct is, I am not sure its a good thing for my current pipeline. GPU memory is a premium for me. I don’t have any GPU memory available to load the next batch and be doing pre-processing while the GPU is doing training. With CPU based augmentations, they are actually super quick,and they use CPU memory. I use pin_memory and a async copies into GPU memory, so my GPU is at 100%. If your GPU is at 100% then its not waiting on you and I can’t imagine GPU augmentations being better. In short, I think CPU + GPU is a powerful combination. For just straight up doing pre-processing, but not online pre-processing, then I think DALI sounds great. Either way, I am going to look into it.", "votes": 2 }, { "id": 972363, "postDate": "2020-08-16T13:43:54.437Z", "content": "i tried it in deepfake competition but no success because of memory leak.
", "rawMarkdown": "i tried it in deepfake competition but no success because of memory leak. ", "votes": 1 } ] } ] }, { "id": 962954, "postDate": "2020-08-08T15:10:17.300Z", "content": "Resnest are fast indeed and give good results :)
\nYou have even faster versions of Resnest50 from the hub
\n'resnest50_fast_1s1x64d', 'resnest50_fast_1s2x40d',\n'resnest50_fast_1s4x24d', 'resnest50_fast_2s1x64d','\nresnest50_fast_2s2x40d', 'resnest50_fast_4s1x64d',\n'resnest50_fast_4s2x40d'\nFor efficientnet , I would recommend using timm by @rwightman instead of this repo
", "rawMarkdown": "Resnest are fast indeed and give good results :)\n\nYou have even faster versions of Resnest50 from the hub\n```\n'resnest50_fast_1s1x64d', 'resnest50_fast_1s2x40d',\n'resnest50_fast_1s4x24d', 'resnest50_fast_2s1x64d','\nresnest50_fast_2s2x40d', 'resnest50_fast_4s1x64d',\n'resnest50_fast_4s2x40d'\n``` \n\n\nFor efficientnet , I would recommend using [timm](https://github.com/rwightman/pytorch-image-models) by @rwightman instead of this repo", "votes": 4, "replies": [ { "id": 962979, "postDate": "2020-08-08T15:23:31.310Z", "content": "Thanks for the tip. Do the fast version deliver the same performance or is there a trade-off in your experience ? The paper suggest lower performance on theirbenchmark but I was wondering what you think based on experience.
\nWill give the efficientnet link a shot :)
", "rawMarkdown": "Thanks for the tip. Do the fast version deliver the same performance or is there a trade-off in your experience ? The paper suggest lower performance on theirbenchmark but I was wondering what you think based on experience.\n\nWill give the efficientnet link a shot :)", "votes": 2 }, { "id": 962996, "postDate": "2020-08-08T15:32:03.433Z", "content": "I tried just the first one, it run slightly faster with similar result.
\nFor timm, you have much more models and weignts than Pytorch-Efficientnet repo. For instance I could get LB 0.952 with B5 noisy Student on timm
Does it run faster or it's just additional weights ? My main issue with B5 is that it takes ages to train locally for me.
", "rawMarkdown": "Does it run faster or it's just additional weights ? My main issue with B5 is that it takes ages to train locally for me.", "votes": 1, "replies": [ { "id": 963016, "postDate": "2020-08-08T15:44:17.907Z", "content": "\n\nDoes it run faster or it's just additional weights ? My main issue with B5 is that it takes ages to train locally for me.
\n
For timm ? my recommendation was more based on options and flexibility but I didn't really compare speeds.
\nFor these big models I use either TPU or Multi-GPU.
", "rawMarkdown": "> Does it run faster or it's just additional weights ? My main issue with B5 is that it takes ages to train locally for me.\n\nFor timm ? my recommendation was more based on options and flexibility but I didn't really compare speeds. \n\nFor these big models I use either TPU or Multi-GPU. ", "votes": 1 }, { "id": 963454, "postDate": "2020-08-09T03:44:29.520Z", "content": "@serigne Hi Serigne may I know how is your TPU usage % when running the pytorch model?
\nDid you apply distributed parallel loader and multiprocessing.spawn as well?
I tried but feel frustrated and at the end switch to tensorflow
\nthanks
", "rawMarkdown": "@serigne Hi Serigne may I know how is your TPU usage % when running the pytorch model? \nDid you apply distributed parallel loader and multiprocessing.spawn as well?\n\nI tried but feel frustrated and at the end switch to tensorflow\n\nthanks" }, { "id": 963627, "postDate": "2020-08-09T07:20:47.333Z", "rawMarkdown": "", "isDeleted": true } ] } ] }, { "id": 970243, "postDate": "2020-08-14T09:40:57.917Z", "content": "Resnest50 converges much faster than Efficientnet and is less prone to noise. But weirdly, I'm getting large CV and LB gap using this model. I'm using resnest50d_1s4x24d. One of my experiments yielded 0.9254 on CV and 0.903 on LB. 😨
It's true my intial kernels in the beginning with resnet50 with 5 folds 128 image size gave a socre of 0.91.
\nEfficientNet in pytorch does run slow. Given Lower score.
Thanks, I think it explains some of my problems with pytorch in this competition.
", "rawMarkdown": "Thanks, I think it explains some of my problems with pytorch in this competition.", "votes": 1 }, { "id": 963867, "postDate": "2020-08-09T11:24:24.457Z", "content": "For TF/Keras user, : ResNeSt , TF version.
But note, no pre-trained weights for tf version yet, one needs to convert it to try.
I have a converted version here but the CV LB gap was large.
", "rawMarkdown": "I have a converted version [here](https://github.com/RichardXiao13/TensorFlow-ResNets) but the CV LB gap was large.", "votes": 2 }, { "id": 972058, "postDate": "2020-08-16T07:41:15.700Z", "content": "@richardxiao03
\nAmazing. Thanks for sharing brother. Starred. -)
I came across this issue a few months ago with pytorch, it's crazy they still didn't optimize it.
", "rawMarkdown": "I came across this issue a few months ago with pytorch, it's crazy they still didn't optimize it.", "votes": 1 }, { "id": 963509, "postDate": "2020-08-09T04:55:16.817Z", "content": "I am trying resnext50 from here. It can easily give .93LB on images of size 256. They are faster than resnest too.
", "rawMarkdown": "I am trying resnext50 from [here](https://pytorch.org/hub/facebookresearch_semi-supervised-ImageNet1K-models_resnext/). It can easily give .93LB on images of size 256. They are faster than resnest too. ", "votes": 1 }, { "id": 965773, "postDate": "2020-08-10T20:58:59.317Z", "content": "I've just submitted my resnest50_fast_1s1x64d model trained on 384*384
LB 0.946 : 5 folds + 15 TTA for each fold inference
\nIt run really Fast on TPU (with big batch size)
", "rawMarkdown": "I've just submitted my `resnest50_fast_1s1x64d` model trained on 384*384\n\nLB 0.946 : 5 folds + 15 TTA for each fold inference \n\nIt run really Fast on TPU (with big batch size) ", "votes": 2, "replies": [ { "id": 965838, "postDate": "2020-08-10T22:32:17.110Z", "content": "I'm getting similar resullts. Just started with google colab pro. Currently running one training through a GPU. Will try TPU after and hope pytorch lightning is able to seamlessly make the transition.
", "rawMarkdown": "I'm getting similar resullts. Just started with google colab pro. Currently running one training through a GPU. Will try TPU after and hope pytorch lightning is able to seamlessly make the transition." }, { "id": 965839, "postDate": "2020-08-10T22:36:36.847Z", "content": "@serigne what is your CV for this 0.946 sub?
", "rawMarkdown": "@serigne what is your CV for this 0.946 sub?" }, { "id": 966050, "postDate": "2020-08-11T05:11:39.460Z", "content": "@cpmpml OOF (with 5 TTA per fold) = 0.930
", "rawMarkdown": "@cpmpml OOF (with 5 TTA per fold) = 0.930", "votes": 1 }, { "id": 966147, "postDate": "2020-08-11T07:24:10.347Z", "content": "Can you share information about the package information from where you are using 'resnest50_fast_1s1x64d'.
", "rawMarkdown": "Can you share information about the package information from where you are using 'resnest50_fast_1s1x64d'. " }, { "id": 966497, "postDate": "2020-08-11T13:13:58.023Z", "content": "I use the official repo
", "rawMarkdown": "I use the [official repo](https://github.com/zhanghang1989/ResNeSt)\n\n", "votes": 1 } ] }, { "id": 966496, "postDate": "2020-08-11T13:13:39.147Z", "content": "All you all using the same head/classifier with these different models, that you were using with EFnets?
\n\nRight now my head is pretty basic for EFnets, any advice for improving it?
\n\n self.meta_fc1 = nn.Linear(n_meta_features, 4)\n self.meta_bn0 = nn.BatchNorm1d(4)\n self.combined_fc0 = nn.Linear(num_ftrs + 4, 1)\n self.avgpool0 = nn.AdaptiveAvgPool2d((1,1))\n self.dropout0 = nn.Dropout(p=0.2)\n\n self.dense = nn.Linear(num_ftrs, 1)\n\ndef forward(self, inputs):\n\n # Image + Meta\n image, meta = inputs\n cnn_features = self.arch.extract_features(image) \n cnn_features = self.avgpool0(cnn_features) \n cnn_features = cnn_features.flatten(start_dim=1)\n meta_features = self.meta_fc1(meta)\n meta_features = F.selu(meta_features)\n meta_features = self.meta_bn0(meta_features)\n features = torch.cat((cnn_features, meta_features), dim=1)\n output = self.combined_fc0(features)\n output = torch.sigmoid(output)\n return output \nHi, you are new to Kaggle hence I feel I should give you a little friendly advice. It is considered bad practice to share specific information during the last week of competition. Why? Because in some cases, people shared very valuable info on the last day and it blew up the leaderboard. Sharing at least a week in advance makes sure every participant has a fair chance to leverage what has been shared.
\nI am not saying you are giving away very valuable info, but you might.
\nDisclaimer: I don't work at Kaggle nor am I tied to Kaggle in anyway, you can freely ignore my friendly advice ;)
", "rawMarkdown": "Hi, you are new to Kaggle hence I feel I should give you a little friendly advice. It is considered bad practice to share specific information during the last week of competition. Why? Because in some cases, people shared very valuable info on the last day and it blew up the leaderboard. Sharing at least a week in advance makes sure every participant has a fair chance to leverage what has been shared.\n\nI am not saying you are giving away very valuable info, but you might.\n\nDisclaimer: I don't work at Kaggle nor am I tied to Kaggle in anyway, you can freely ignore my friendly advice ;)\n", "votes": 3 } ] } ], "comments": [ { "id": 964937, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-10T09:25:41.800000", "content": "First try isn't good, I guess I have to retune everything.
", "votes": 3, "replies": [] }, { "id": 963322, "author_name": "Iafoss", "author_url": "", "post_date": "2020-08-08T22:32:50.793000", "content": "\n\nAll this is of course slower than TF + TPU :)
\n
It's a little bit sad that Pytorch doesn't have a proper way right now to load data directly on TPUs( need to use CPU -> Image data preprocessing is extremely slow at kaggle. So XLA could be really used at kaggle only for NLP, and even their 16 GB RAM limitation makes it difficult to deal with. While TF is not flexible, and it is difficult to add even simple things.
", "votes": 3, "replies": [ { "id": 963334, "author_name": "Arnaud Roussel", "author_url": "", "post_date": "2020-08-08T23:03:00.687000", "content": "Yes, exactly my thoughts. If at least there was an equivalent to fast I/O + processing I'd gladly invest modifying my code to use XLA (or use pytorch lightning as I do locally but usually lightning for Kaggle TPU often fails for me).
", "votes": 1, "replies": [] }, { "id": 963342, "author_name": "Serigne ", "author_url": "", "post_date": "2020-08-08T23:33:27.493000", "content": "I/O + processing is clearly an issue, however Colab Pro has much better config for Pytorch TPU than Kaggle. More CPU cores and +35 GB of RAM.
\nAnd you can run it h24
@serigne is it available in France? So far only available in the US, right?
", "votes": 0, "replies": [] }, { "id": 964031, "author_name": "Serigne ", "author_url": "", "post_date": "2020-08-09T14:28:48.907000", "content": "Yeah it's available. Don't worry about the US zip code, you can put yours.
", "votes": 2, "replies": [] }, { "id": 964273, "author_name": "PAB97", "author_url": "", "post_date": "2020-08-09T17:55:04.747000", "content": "Alright thanks!
", "votes": 0, "replies": [] }, { "id": 965539, "author_name": "Arnaud Roussel", "author_url": "", "post_date": "2020-08-10T17:47:22.077000", "content": "@serigne Thanks for the colab pro tip. Didn't know that service existed. Price is pretty good too. Using it know for bigger model.
", "votes": 0, "replies": [] }, { "id": 966057, "author_name": "Serigne ", "author_url": "", "post_date": "2020-08-11T05:23:10.470000", "content": "De rien @arroqc
\nNote that on TPU , you can execute a notebook, close the browser and il will be still running. You can open it hours later elsewhere and find all the training log.
\nThis doesn't work for GPU though. It would disconnect after about 1 hour of no interactive session.
", "votes": 0, "replies": [] }, { "id": 972018, "author_name": "Chandan Verma", "author_url": "", "post_date": "2020-08-16T07:02:18.087000", "content": "Have you tried NVIDIA-DALI which uses GPU for image augmentations and loads images directly on to GPU which makes data loading much faster as compared to the traditional approach.
", "votes": 5, "replies": [ { "id": 972062, "author_name": "Signal", "author_url": "", "post_date": "2020-08-16T07:43:10.140000", "content": "@chandanverma I personally have not tried DALI, but I am going to look into it because that tech sounds interesting. However, my instinct is, I am not sure its a good thing for my current pipeline. GPU memory is a premium for me. I don’t have any GPU memory available to load the next batch and be doing pre-processing while the GPU is doing training. With CPU based augmentations, they are actually super quick,and they use CPU memory. I use pin_memory and a async copies into GPU memory, so my GPU is at 100%. If your GPU is at 100% then its not waiting on you and I can’t imagine GPU augmentations being better. In short, I think CPU + GPU is a powerful combination. For just straight up doing pre-processing, but not online pre-processing, then I think DALI sounds great. Either way, I am going to look into it.
", "votes": 2, "replies": [] }, { "id": 972363, "author_name": "yimacs", "author_url": "", "post_date": "2020-08-16T13:43:54.437000", "content": "i tried it in deepfake competition but no success because of memory leak.
", "votes": 1, "replies": [] } ] } ] }, { "id": 962954, "author_name": "Serigne ", "author_url": "", "post_date": "2020-08-08T15:10:17.300000", "content": "Resnest are fast indeed and give good results :)
\nYou have even faster versions of Resnest50 from the hub
\n'resnest50_fast_1s1x64d', 'resnest50_fast_1s2x40d',\n'resnest50_fast_1s4x24d', 'resnest50_fast_2s1x64d','\nresnest50_fast_2s2x40d', 'resnest50_fast_4s1x64d',\n'resnest50_fast_4s2x40d'\nFor efficientnet , I would recommend using timm by @rwightman instead of this repo
", "votes": 4, "replies": [ { "id": 962979, "author_name": "Arnaud Roussel", "author_url": "", "post_date": "2020-08-08T15:23:31.310000", "content": "Thanks for the tip. Do the fast version deliver the same performance or is there a trade-off in your experience ? The paper suggest lower performance on theirbenchmark but I was wondering what you think based on experience.
\nWill give the efficientnet link a shot :)
", "votes": 2, "replies": [] }, { "id": 962996, "author_name": "Serigne ", "author_url": "", "post_date": "2020-08-08T15:32:03.433000", "content": "I tried just the first one, it run slightly faster with similar result.
\nFor timm, you have much more models and weignts than Pytorch-Efficientnet repo. For instance I could get LB 0.952 with B5 noisy Student on timm
Does it run faster or it's just additional weights ? My main issue with B5 is that it takes ages to train locally for me.
", "votes": 1, "replies": [ { "id": 963016, "author_name": "Serigne ", "author_url": "", "post_date": "2020-08-08T15:44:17.907000", "content": "\n\nDoes it run faster or it's just additional weights ? My main issue with B5 is that it takes ages to train locally for me.
\n
For timm ? my recommendation was more based on options and flexibility but I didn't really compare speeds.
\nFor these big models I use either TPU or Multi-GPU.
", "votes": 1, "replies": [] }, { "id": 963454, "author_name": "Salaryman", "author_url": "", "post_date": "2020-08-09T03:44:29.520000", "content": "@serigne Hi Serigne may I know how is your TPU usage % when running the pytorch model?
\nDid you apply distributed parallel loader and multiprocessing.spawn as well?
I tried but feel frustrated and at the end switch to tensorflow
\nthanks
", "votes": 0, "replies": [] }, { "id": 963627, "author_name": "", "author_url": "", "post_date": "2020-08-09T07:20:47.333000", "content": "", "votes": 0, "replies": [] } ] } ] }, { "id": 970243, "author_name": "Tahsin Mostafiz", "author_url": "", "post_date": "2020-08-14T09:40:57.917000", "content": "Resnest50 converges much faster than Efficientnet and is less prone to noise. But weirdly, I'm getting large CV and LB gap using this model. I'm using resnest50d_1s4x24d. One of my experiments yielded 0.9254 on CV and 0.903 on LB. 😨
It's true my intial kernels in the beginning with resnet50 with 5 folds 128 image size gave a socre of 0.91.
\nEfficientNet in pytorch does run slow. Given Lower score.
Thanks, I think it explains some of my problems with pytorch in this competition.
", "votes": 1, "replies": [] }, { "id": 963867, "author_name": "Innat", "author_url": "", "post_date": "2020-08-09T11:24:24.457000", "content": "For TF/Keras user, : ResNeSt , TF version.
But note, no pre-trained weights for tf version yet, one needs to convert it to try.
I have a converted version here but the CV LB gap was large.
", "votes": 2, "replies": [] }, { "id": 972058, "author_name": "Innat", "author_url": "", "post_date": "2020-08-16T07:41:15.700000", "content": "@richardxiao03
\nAmazing. Thanks for sharing brother. Starred. -)
I came across this issue a few months ago with pytorch, it's crazy they still didn't optimize it.
", "votes": 1, "replies": [] }, { "id": 963509, "author_name": "Vishnu Subramanian", "author_url": "", "post_date": "2020-08-09T04:55:16.817000", "content": "I am trying resnext50 from here. It can easily give .93LB on images of size 256. They are faster than resnest too.
", "votes": 1, "replies": [] }, { "id": 965773, "author_name": "Serigne ", "author_url": "", "post_date": "2020-08-10T20:58:59.317000", "content": "I've just submitted my resnest50_fast_1s1x64d model trained on 384*384
LB 0.946 : 5 folds + 15 TTA for each fold inference
\nIt run really Fast on TPU (with big batch size)
", "votes": 2, "replies": [ { "id": 965838, "author_name": "Arnaud Roussel", "author_url": "", "post_date": "2020-08-10T22:32:17.110000", "content": "I'm getting similar resullts. Just started with google colab pro. Currently running one training through a GPU. Will try TPU after and hope pytorch lightning is able to seamlessly make the transition.
", "votes": 0, "replies": [] }, { "id": 965839, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-10T22:36:36.847000", "content": "@serigne what is your CV for this 0.946 sub?
", "votes": 0, "replies": [] }, { "id": 966050, "author_name": "Serigne ", "author_url": "", "post_date": "2020-08-11T05:11:39.460000", "content": "@cpmpml OOF (with 5 TTA per fold) = 0.930
", "votes": 1, "replies": [] }, { "id": 966147, "author_name": "Vishnu Subramanian", "author_url": "", "post_date": "2020-08-11T07:24:10.347000", "content": "Can you share information about the package information from where you are using 'resnest50_fast_1s1x64d'.
", "votes": 0, "replies": [] }, { "id": 966497, "author_name": "Serigne ", "author_url": "", "post_date": "2020-08-11T13:13:58.023000", "content": "I use the official repo
", "votes": 1, "replies": [] } ] }, { "id": 966496, "author_name": "Signal", "author_url": "", "post_date": "2020-08-11T13:13:39.147000", "content": "All you all using the same head/classifier with these different models, that you were using with EFnets?
\n\nRight now my head is pretty basic for EFnets, any advice for improving it?
\n\n self.meta_fc1 = nn.Linear(n_meta_features, 4)\n self.meta_bn0 = nn.BatchNorm1d(4)\n self.combined_fc0 = nn.Linear(num_ftrs + 4, 1)\n self.avgpool0 = nn.AdaptiveAvgPool2d((1,1))\n self.dropout0 = nn.Dropout(p=0.2)\n\n self.dense = nn.Linear(num_ftrs, 1)\n\ndef forward(self, inputs):\n\n # Image + Meta\n image, meta = inputs\n cnn_features = self.arch.extract_features(image) \n cnn_features = self.avgpool0(cnn_features) \n cnn_features = cnn_features.flatten(start_dim=1)\n meta_features = self.meta_fc1(meta)\n meta_features = F.selu(meta_features)\n meta_features = self.meta_bn0(meta_features)\n features = torch.cat((cnn_features, meta_features), dim=1)\n output = self.combined_fc0(features)\n output = torch.sigmoid(output)\n return output \nHi, you are new to Kaggle hence I feel I should give you a little friendly advice. It is considered bad practice to share specific information during the last week of competition. Why? Because in some cases, people shared very valuable info on the last day and it blew up the leaderboard. Sharing at least a week in advance makes sure every participant has a fair chance to leverage what has been shared.
\nI am not saying you are giving away very valuable info, but you might.
\nDisclaimer: I don't work at Kaggle nor am I tied to Kaggle in anyway, you can freely ignore my friendly advice ;)
", "votes": 3, "replies": [] } ] } ], "raw_markdown_by_id": { "962929": "It's a known issue with efficient nets that it is usually slower on Pytorch. Afaik depthwise convolution are really poorly optimized in Pytorch as can be seen in some tickets both on the efficientnet repo: https://github.com/lukemelas/EfficientNet-PyTorch/issues/159\nand pytorch: https://github.com/pytorch/pytorch/issues/18631\n\nThankfully there is a recent paper proposing ResNest https://arxiv.org/pdf/2004.08955.pdf and a pytorch implementation here https://github.com/zhanghang1989/ResNeSt. For example for me a B2 and a Resnest50 train in approx the same time but the resnest50 showed a good CV/LB increase since it's a bigger model at 25M instead of the 7M for the B2.\n\nSome numbers on my gtx1080ti:\nB2 7M 7:30 per epoch\nResnest50 25M 6:45 per epoch\nB4 19M 20:00 per epoch\nResnest100 46M 14:00 per epoch\n\nAll this is of course slower than TF + TPU :)", "964937": "First try isn't good, I guess I have to retune everything.", "963322": "> All this is of course slower than TF + TPU :)\n\nIt's a little bit sad that Pytorch doesn't have a proper way right now to load data directly on TPUs( need to use CPU -> Image data preprocessing is extremely slow at kaggle. So XLA could be really used at kaggle only for NLP, and even their 16 GB RAM limitation makes it difficult to deal with. While TF is not flexible, and it is difficult to add even simple things.\n", "962954": "Resnest are fast indeed and give good results :)\n\nYou have even faster versions of Resnest50 from the hub\n```\n'resnest50_fast_1s1x64d', 'resnest50_fast_1s2x40d',\n'resnest50_fast_1s4x24d', 'resnest50_fast_2s1x64d','\nresnest50_fast_2s2x40d', 'resnest50_fast_4s1x64d',\n'resnest50_fast_4s2x40d'\n``` \n\n\nFor efficientnet , I would recommend using [timm](https://github.com/rwightman/pytorch-image-models) by @rwightman instead of this repo", "970243": "`Resnest50` converges much faster than Efficientnet and is less prone to noise. But weirdly, I'm getting large CV and LB gap using this model. I'm using `resnest50d_1s4x24d`. One of my experiments yielded `0.9254` on CV and `0.903` on LB. 😨", "965368": "It's true my intial kernels in the beginning with resnet50 with 5 folds 128 image size gave a socre of 0.91.\nEfficientNet in pytorch does run slow. Given Lower score.", "964305": "Thanks, I think it explains some of my problems with pytorch in this competition.", "963867": "For `TF/Keras` user, : [ResNeSt](https://github.com/QiaoranC/tf_ResNeSt_RegNet_model) , TF version.\n\nBut note, no pre-trained weights for `tf` version yet, one needs to [convert](https://github.com/nerox8664/pytorch2keras) it to try. ", "963618": "I came across this issue a few months ago with pytorch, it's crazy they still didn't optimize it.", "963509": "I am trying resnext50 from [here](https://pytorch.org/hub/facebookresearch_semi-supervised-ImageNet1K-models_resnext/). It can easily give .93LB on images of size 256. They are faster than resnest too. ", "965773": "I've just submitted my `resnest50_fast_1s1x64d` model trained on 384*384\n\nLB 0.946 : 5 folds + 15 TTA for each fold inference \n\nIt run really Fast on TPU (with big batch size) ", "966496": "All you all using the same head/classifier with these different models, that you were using with EFnets?\n\nRight now my head is pretty basic for EFnets, any advice for improving it?\n\n self.meta_fc1 = nn.Linear(n_meta_features, 4)\n self.meta_bn0 = nn.BatchNorm1d(4)\n self.combined_fc0 = nn.Linear(num_ftrs + 4, 1)\n self.avgpool0 = nn.AdaptiveAvgPool2d((1,1))\n self.dropout0 = nn.Dropout(p=0.2)\n \n self.dense = nn.Linear(num_ftrs, 1)\n \n def forward(self, inputs):\n\n # Image + Meta\n image, meta = inputs\n cnn_features = self.arch.extract_features(image) \n cnn_features = self.avgpool0(cnn_features) \n cnn_features = cnn_features.flatten(start_dim=1)\n meta_features = self.meta_fc1(meta)\n meta_features = F.selu(meta_features)\n meta_features = self.meta_bn0(meta_features)\n features = torch.cat((cnn_features, meta_features), dim=1)\n output = self.combined_fc0(features)\n output = torch.sigmoid(output)\n return output \n " } }