{ "id": 168152, "title": "What's your best (single) Pytorch Model score ?", "url": "/competitions/siim-isic-melanoma-classification/discussion/168152", "author_name": "Serigne ", "post_date": "2020-07-19T13:00:42.957000", "votes": 37, "comment_count": 111, "views": 0, "content": "

I know one can achieve +0.955 on LB by tweaking a bit some great public TF kernels like this one or this one. But for now I want to experiment as much as possible with Pytorch.

\n

After many exepriments with Pytorch, I could achieve 0.944 with 5 folds efficientnetB5 noisy student with some heavy augmentations and trained on @cdeotte 384*384 jpeg files (with external data)

\n

I wonder if someone at the top managed to achieve +0.95/+0.96 with a Pytorch model ?

\n

Are TPU + TFRecords ( allowing huge models + big resolutions and big batch_size trained in short amount of time) the only way to achieve +0.96 ? :)

\n

Edit : the model achieve 0.9496 on LB with 15 TTA for each fold.

", "messages": [ { "id": 935509, "postDate": "2020-07-19T13:00:42.957Z", "content": "

I know one can achieve +0.955 on LB by tweaking a bit some great public TF kernels like this one or this one. But for now I want to experiment as much as possible with Pytorch.

\n

After many exepriments with Pytorch, I could achieve 0.944 with 5 folds efficientnetB5 noisy student with some heavy augmentations and trained on @cdeotte 384*384 jpeg files (with external data)

\n

I wonder if someone at the top managed to achieve +0.95/+0.96 with a Pytorch model ?

\n

Are TPU + TFRecords ( allowing huge models + big resolutions and big batch_size trained in short amount of time) the only way to achieve +0.96 ? :)

\n

Edit : the model achieve 0.9496 on LB with 15 TTA for each fold.

", "rawMarkdown": "I know one can achieve +0.955 on LB by tweaking a bit some great public TF kernels like [this one](https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords) or [this one](https://www.kaggle.com/agentauers/incredible-tpus-finetune-effnetb0-b6-at-once). But for now I want to experiment as much as possible with Pytorch. \n\nAfter many exepriments with Pytorch, I could achieve 0.944 with 5 folds efficientnetB5 noisy student with some heavy augmentations and trained on @cdeotte 384*384 jpeg files (with external data)\n\nI wonder if someone at the top managed to achieve +0.95/+0.96 with a Pytorch model ? \n\nAre TPU + TFRecords ( allowing huge models + big resolutions and big batch_size trained in short amount of time) the only way to achieve +0.96 ? :)\n\n\nEdit : the model achieve 0.9496 on LB with 15 TTA for each fold. \n", "votes": 37 }, { "id": 939957, "postDate": "2020-07-22T15:36:12.797Z", "content": "

I got >0.95 with a single model which is of similar size as your model and after adding some sauce to it (model wise, no LB probing) I got 0.96 from a single model. But you should not take every LB result as is, the test set is small and very un-balanced with very low positive examples. a high Public LB score can turn into a low Private LB. If I judge by my CV std - Shakeup is Coming

", "rawMarkdown": "I got >0.95 with a single model which is of similar size as your model and after adding some sauce to it (model wise, no LB probing) I got 0.96 from a single model. But you should not take every LB result as is, the test set is small and very un-balanced with very low positive examples. a high Public LB score can turn into a low Private LB. If I judge by my CV std - **Shakeup is Coming** ", "votes": 14, "replies": [ { "id": 939969, "postDate": "2020-07-22T15:48:03.497Z", "content": "

0.96 by single model! 💥
\nThat sauce must be damn tasty!

", "rawMarkdown": "0.96 by single model! 💥\nThat sauce must be damn tasty! ", "votes": 1 }, { "id": 940020, "postDate": "2020-07-22T16:17:58.943Z", "content": "

Thanks for sharing!

\n

Have you been able to reproduce your single score >0.96? or it's been a one shot?

", "rawMarkdown": "Thanks for sharing!\n\nHave you been able to reproduce your single score >0.96? or it's been a one shot?", "votes": 1 }, { "id": 940130, "postDate": "2020-07-22T17:27:10.670Z", "content": "

Thanks @yuval6967 for the feedback!\nI will definitely trust my CV

", "rawMarkdown": "Thanks @yuval6967 for the feedback!\nI will definitely trust my CV", "votes": 1 }, { "id": 940331, "postDate": "2020-07-22T20:46:46.500Z", "content": "

@optimo I don't know about @yuval6967 but I'm having deterministic results with Pytorch.

\n\n

It's impossible with Tensorflow

", "rawMarkdown": "@optimo I don't know about @yuval6967 but I'm having deterministic results with Pytorch. \n\nIt's impossible with Tensorflow", "votes": 1 }, { "id": 940336, "postDate": "2020-07-22T20:56:47.463Z", "content": "

Well @serigne yes I meant reproduce the same result with a different random seed, independently of being able to fix everything. Would this model produce above 0.96 with any random seed?

", "rawMarkdown": "Well @serigne yes I meant reproduce the same result with a different random seed, independently of being able to fix everything. Would this model produce above 0.96 with any random seed?" }, { "id": 940339, "postDate": "2020-07-22T21:04:53.380Z", "content": "

Random seed is a feature as another. Of course you may have different results by changing it.

\n\n

But the difference should be small to make sure the model is stable.

", "rawMarkdown": "Random seed is a feature as another. Of course you may have different results by changing it. \n\nBut the difference should be small to make sure the model is stable. " }, { "id": 941857, "postDate": "2020-07-23T13:02:27.640Z", "content": "

@optimo yes I was able to get a similar results with a slightly different model and different training parameters + different seed. As the LB is very noisy, the result won't be reproduced with any random seed, my CV std is ~0.004 (between folds). (I stopped aiming for the highest Public LB long ago as I don't really believe in it)

", "rawMarkdown": "@optimo yes I was able to get a similar results with a slightly different model and different training parameters + different seed. As the LB is very noisy, the result won't be reproduced with any random seed, my CV std is ~0.004 (between folds). (I stopped aiming for the highest Public LB long ago as I don't really believe in it) ", "votes": 2 }, { "id": 941972, "postDate": "2020-07-23T14:22:40.843Z", "content": "

@yuval6967 thanks! I agree, it takes some wisdom to refrain from climbing Public LB... but Private LB will bring some nice reward! Good luck!

", "rawMarkdown": "@yuval6967 thanks! I agree, it takes some wisdom to refrain from climbing Public LB... but Private LB will bring some nice reward! Good luck!", "votes": 1 } ] }, { "id": 946965, "postDate": "2020-07-27T01:41:33.627Z", "content": "

Note: Everyone is reporting their CV here. We should also report how our CV is setup because without knowing what data people are adding to their validation folds and whether people are using triple stratified or not, we cannot compare CVs.

\n\n

For example, if you use external data, don't stratify, don't remove duplicate images, and allow external data in your validation fold, it is easy to get 5 KFold CV AUC 0.98+

", "rawMarkdown": "Note: Everyone is reporting their CV here. We should also report how our CV is setup because without knowing what data people are adding to their validation folds and whether people are using triple stratified or not, we cannot compare CVs.\n\nFor example, if you use external data, don't stratify, don't remove duplicate images, and allow external data in your validation fold, it is easy to get 5 KFold CV AUC 0.98+ \n\n[1]: https://www.kaggle.com/nroman/melanoma-pytorch-starter-efficientnet?scriptVersionId=35185525", "votes": 6, "replies": [ { "id": 947368, "postDate": "2020-07-27T08:07:53.797Z", "content": "

Using external data in validation doesn't make sense indeed.

\n\n

What I did, was to create firstly K-fold from your Triplet Stratified CSV of 2020 data. Then I added a part of external data on the training part for each fold.
\nhowever I still noticed huge gap between folds or between some folds and their corresponding LB submission. That why I don't rely very much on regular CV and try instead Snapshot training and multiple checkpoints averaging.

", "rawMarkdown": "Using external data in validation doesn't make sense indeed. \n\nWhat I did, was to create firstly K-fold from your Triplet Stratified CSV of 2020 data. Then I added a part of external data on the training part for each fold. \nhowever I still noticed huge gap between folds or between some folds and their corresponding LB submission. That why I don't rely very much on regular CV and try instead Snapshot training and multiple checkpoints averaging. ", "votes": 1 }, { "id": 947400, "postDate": "2020-07-27T08:43:18.757Z", "content": "

@serigne you are right about variations in CV of each fold, in my case my per fold CV varies from 0.92-0.94

", "rawMarkdown": "@serigne you are right about variations in CV of each fold, in my case my per fold CV varies from 0.92-0.94", "votes": 1 }, { "id": 948288, "postDate": "2020-07-27T19:27:58.290Z", "content": "

I guess it's also worth mentioning that taking the mean of the k AUC scores from each fold is not equal to calculating the global AUC from the out of fold predictions

", "rawMarkdown": "I guess it's also worth mentioning that taking the mean of the k AUC scores from each fold is not equal to calculating the global AUC from the out of fold predictions", "votes": 1 }, { "id": 948290, "postDate": "2020-07-27T19:30:03.900Z", "content": "

spoiler alert: using external data in validation is one of the largest failures which people do in this competition

", "rawMarkdown": "spoiler alert: using external data in validation is one of the largest failures which people do in this competition", "votes": 2 }, { "id": 948354, "postDate": "2020-07-27T21:00:25.407Z", "content": "

@jacekpoplawski I don't know about that, but certainly avoiding it is easy.

\n\n

Say your train/non-external data goes up to index 32000. Then you concatenate external data 1, external data 2, external data 3......etc. Now you do your splits/folds (using Scikit or whatever). You now have list of train_indexes and val_indexes. Simply do the following to the val_indexes at the top of each fold:

\n\n

```\nval_indexes = val_indexes[val_indexes <= 32000]

\n\n

```\nNote, I am assuming here that val_indexes is a Numpy array (which is what Scikit produces).

", "rawMarkdown": "@jacekpoplawski I don't know about that, but certainly avoiding it is easy.\n\nSay your train/non-external data goes up to index 32000. Then you concatenate external data 1, external data 2, external data 3......etc. Now you do your splits/folds (using Scikit or whatever). You now have list of `train_indexes` and `val_indexes`. Simply do the following to the `val_indexes` at the top of each fold:\n\n```\nval_indexes = val_indexes[val_indexes <= 32000]\n\n```\nNote, I am assuming here that val_indexes is a Numpy array (which is what Scikit produces). \n\n" } ] }, { "id": 974360, "postDate": "2020-08-17T23:42:35.297Z", "content": "

Could not tune effnet b5, my best is b4 with LB 0.9474, and a bit over 95 with a blend.

\n

I did not blend with any public kernels, couldn't resolve to it. There is a limit to what I am ready to do to get Kaggle points, LOL.

\n

I regret I forgot to try two ideas I had but5 overall it was a great learning experience. Maybe 10 years form now I'll be competitive in computer vision ;) Maybe I should just buy code, I see some gave pointers to relevant online stores…

", "rawMarkdown": "Could not tune effnet b5, my best is b4 with LB 0.9474, and a bit over 95 with a blend.\n\nI did not blend with any public kernels, couldn't resolve to it. There is a limit to what I am ready to do to get Kaggle points, LOL.\n\nI regret I forgot to try two ideas I had but5 overall it was a great learning experience. Maybe 10 years form now I'll be competitive in computer vision ;) Maybe I should just buy code, I see some gave pointers to relevant online stores...", "votes": 3, "replies": [ { "id": 974371, "postDate": "2020-08-17T23:50:54.800Z", "content": "

@cpmpml I can respect that. I did not blend any public kernel either and I am right where you are. Integrity is one of the if not the most important characteristic in Science. If I medal it is in no doubt due to many contributions shared by many people, but those contributions are not going to be scores that I \"blend\".

", "rawMarkdown": "@cpmpml I can respect that. I did not blend any public kernel either and I am right where you are. Integrity is one of the if not the most important characteristic in Science. If I medal it is in no doubt due to many contributions shared by many people, but those contributions are not going to be scores that I \"blend\".", "votes": 1 }, { "id": 974380, "postDate": "2020-08-17T23:59:01.187Z", "content": "

Yes, we seem to have very similar paths in this comp. Your code snippets helped me a lot, thanks again.

", "rawMarkdown": "Yes, we seem to have very similar paths in this comp. Your code snippets helped me a lot, thanks again.", "votes": 1 } ] }, { "id": 972452, "postDate": "2020-08-16T15:06:06.943Z", "content": "

effnetb4, images only, tta, CV 9440 LB 9452

\n

I started with b5 but CV and LB are similar, a bit lower. Trying to tune for a last run …

\n

I think my final rank will be similar to my current rank.

", "rawMarkdown": "effnetb4, images only, tta, CV 9440 LB 9452\n\nI started with b5 but CV and LB are similar, a bit lower. Trying to tune for a last run ...\n\nI think my final rank will be similar to my current rank.\n\n", "votes": 3, "replies": [ { "id": 972463, "postDate": "2020-08-16T15:10:25.363Z", "content": "

Nice job, great CV

", "rawMarkdown": "Nice job, great CV", "votes": 1 }, { "id": 972494, "postDate": "2020-08-16T15:28:48.723Z", "content": "

Thanks Chris. Great CV and poor LB ;)

", "rawMarkdown": "Thanks Chris. Great CV and poor LB ;)\n" }, { "id": 972770, "postDate": "2020-08-16T20:12:30.557Z", "content": "

That is a very good CV.

", "rawMarkdown": "That is a very good CV.", "votes": 1 } ] }, { "id": 935732, "postDate": "2020-07-19T15:36:38.050Z", "content": "

0.9458 effnet-b2, 512x512, 5 fold with 20xTTA, ext data + metadata, local CV 0.925

\n

@serigne have you tried TTA? if you have some heavy augmentation it could work.

", "rawMarkdown": "0.9458 effnet-b2, 512x512, 5 fold with 20xTTA, ext data + metadata, local CV 0.925\n\n@serigne have you tried TTA? if you have some heavy augmentation it could work.", "votes": 3, "replies": [ { "id": 936059, "postDate": "2020-07-19T23:58:25.247Z", "content": "

which TTAs are you referring to?

", "rawMarkdown": "which TTAs are you referring to?" }, { "id": 936303, "postDate": "2020-07-20T06:20:23.997Z", "content": "

Just leaving my training augmentation ON for the test predictions

", "rawMarkdown": "Just leaving my training augmentation ON for the test predictions " } ] }, { "id": 937048, "postDate": "2020-07-20T17:31:11.233Z", "content": "

0.9424 on LB and 0.9407 on Validation\nSingle Eff B5 Only\nImage Dimension 456x456. Used external data\nPytorch

", "rawMarkdown": "0.9424 on LB and 0.9407 on Validation\nSingle Eff B5 Only\nImage Dimension 456x456. Used external data\nPytorch", "votes": 4, "replies": [ { "id": 938542, "postDate": "2020-07-21T15:44:14.973Z", "content": "

May I ask how you tighten the gap between best pytorch model 0.9424 and a score of 95.5 on LB?

\n

A- better single model in tensorflow?
\nB- ensembling of multiple models (no public)?
\nC- ensembling with public notebooks?
\nD- A mix of all this

\n

I'm asking this because I'm very worried that everyone is currently overfiting public LB because of lucky public seeds, so A-B -> I'll try hard more C-D -> I stop trying to climb the LB

", "rawMarkdown": "May I ask how you tighten the gap between best pytorch model 0.9424 and a score of 95.5 on LB?\n\nA- better single model in tensorflow?\nB- ensembling of multiple models (no public)?\nC- ensembling with public notebooks?\nD- A mix of all this\n\nI'm asking this because I'm very worried that everyone is currently overfiting public LB because of lucky public seeds, so A-B -> I'll try hard more C-D -> I stop trying to climb the LB", "votes": 1 }, { "id": 938601, "postDate": "2020-07-21T16:15:26.823Z", "content": "

@optimo for me, the same model with different seeds score : 0.9531 and 0.9502

", "rawMarkdown": "@optimo for me, the **same model** with different seeds score : 0.9531 and 0.9502", "votes": 1 }, { "id": 938642, "postDate": "2020-07-21T16:41:49.030Z", "content": "

Well you seem to be in another planet at the moment^^ congrats!

\n

But the same question could apply to you: is your best model scoring \"only\" 0.9531? Then how do you climb up to 0.9644?

\n

I don't want a secret recipe just to be sure that everyone is not stacking themselves to the same notebooks that scores well on public LB. Are you using any public notebook in your ensemble? (I definitely hope that in order to go top 3 at the moment you are not using a public blender with custom weights but that would be a great info!)

\n

But I guess my question is more towards people ranking from 600th to 50th, as I'm definitely sure that the shake up will be more severe in these positions.

\n

Thanks!

", "rawMarkdown": "Well you seem to be in another planet at the moment^^ congrats!\n\nBut the same question could apply to you: is your best model scoring \"only\" 0.9531? Then how do you climb up to 0.9644?\n\nI don't want a secret recipe just to be sure that everyone is not stacking themselves to the same notebooks that scores well on public LB. Are you using any public notebook in your ensemble? (I definitely hope that in order to go top 3 at the moment you are not using a public blender with custom weights but that would be a great info!)\n\nBut I guess my question is more towards people ranking from 600th to 50th, as I'm definitely sure that the shake up will be more severe in these positions.\n\nThanks!", "votes": 1 }, { "id": 939437, "postDate": "2020-07-22T08:45:11.973Z", "content": "

@optimo I have used this dataset. It comes with multiple folds and I used them. I have been thinking of switching to Triple Stratified Leak-Free KFold CV because it seems more logical. And all the credits for achieving 95.5+ on public LB go to my teammate @ipythonx. He is really good at ensembling. But we are also very much concerned about overfitting the public LB.

", "rawMarkdown": "@optimo I have used [this](https://www.kaggle.com/shonenkov/melanoma-merged-external-data-512x512-jpeg) dataset. It comes with multiple folds and I used them. I have been thinking of switching to [Triple Stratified Leak-Free KFold CV](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/165526) because it seems more logical. And all the credits for achieving `95.5+` on public LB go to my teammate @ipythonx. He is really good at ensembling. But we are also very much concerned about overfitting the public LB." }, { "id": 939498, "postDate": "2020-07-22T09:12:29.817Z", "content": "

thanks @tahsin it might turn out to be an ensemble war then! ^^

", "rawMarkdown": "thanks @tahsin it might turn out to be an ensemble war then! ^^", "votes": 1 }, { "id": 939505, "postDate": "2020-07-22T09:20:20.513Z", "content": "

I think so too. Try to get as much as possible 95+ scores with single models. Then ensemble and pray.

", "rawMarkdown": "I think so too. Try to get as much as possible `95+` scores with single models. Then ensemble and pray. " }, { "id": 943341, "postDate": "2020-07-24T09:56:06.380Z", "content": "

@tahsin I am also getting LB score around 95.8 but very much concerned about overfitting,, Like discuss what techniques are you planning to avoid overfitting.

", "rawMarkdown": "@tahsin I am also getting LB score around 95.8 but very much concerned about overfitting,, Like discuss what techniques are you planning to avoid overfitting. " }, { "id": 943359, "postDate": "2020-07-24T10:05:56.057Z", "content": "

I think he mentioned praying! 🙏 ^^

", "rawMarkdown": "I think he mentioned praying! 🙏 ^^", "votes": 3 } ] }, { "id": 936694, "postDate": "2020-07-20T13:01:41.417Z", "content": "

Thanks @optimo

\n\n

I have run 15 TTA for one fold ..

\n\n

it improved the score from 0.9293 to 0.9387.

\n\n

I will update the overall score once it's done for all 5 folds.

", "rawMarkdown": "Thanks @optimo \n\nI have run 15 TTA for one fold ..\n\nit improved the score from 0.9293 to 0.9387. \n\nI will update the overall score once it's done for all 5 folds. ", "votes": 4, "replies": [ { "id": 937829, "postDate": "2020-07-21T07:22:16.970Z", "content": "

15TTA.......

", "rawMarkdown": "15TTA......." }, { "id": 937962, "postDate": "2020-07-21T09:13:49.343Z", "content": "

It tooks me about 40 minutes to run for each fold on GPU , but the score improvement worth it :)

\n

I will like parallelize it on 8 cores TPU for the next expeiments.

", "rawMarkdown": "It tooks me about 40 minutes to run for each fold on GPU , but the score improvement worth it :)\n\nI will like parallelize it on 8 cores TPU for the next expeiments. " }, { "id": 938546, "postDate": "2020-07-21T15:45:42.367Z", "content": "

so @serigne what did it give you with 5 folds?

", "rawMarkdown": "so @serigne what did it give you with 5 folds?" }, { "id": 938780, "postDate": "2020-07-21T18:33:42.970Z", "content": "

0.9496 on LB

\n

I'm running 512x512 now on B5. I think it will give better score

", "rawMarkdown": "0.9496 on LB \n\nI'm running 512x512 now on B5. I think it will give better score", "votes": 2 }, { "id": 947181, "postDate": "2020-07-27T05:49:14.453Z", "content": "

@serigne what's your CV for 0.9496 LB?

", "rawMarkdown": "@serigne what's your CV for 0.9496 LB?" } ] }, { "id": 974507, "postDate": "2020-08-18T01:00:24.247Z", "content": "

@serigne congrats on the good finish.

", "rawMarkdown": "@serigne congrats on the good finish.", "votes": 1, "replies": [ { "id": 974568, "postDate": "2020-08-18T01:34:19.457Z", "content": "

Thank you very much @cpmpml

\n

\"Always trust you CV\" as they say
\n\"CaptureCV\"
\nI knew my best public LB was heavily overfitted , So I didn't even try to choose it, despite given 3 choices.

\n

My chosen private scored \"just\" 0.9594 on public LB

\n

Even though I have also a not chosen gold solution (private LB 0.9467), it was not my best model. Chosing it would be just luck and not systematic approach. So I am pretty happy with mu current result :)
\n\"capturebest\"

", "rawMarkdown": "Thank you very much @cpmpml \n\n\"Always trust you CV\" as they say \n![CaptureCV](https://i.ibb.co/BsXN56b/Capture-CV.png)\nI knew my best public LB was heavily overfitted , So I didn't even try to choose it, despite given 3 choices. \n\nMy chosen private scored \"just\" 0.9594 on public LB\n\n\nEven though I have also a not chosen gold solution (private LB 0.9467), it was not my best model. Chosing it would be just luck and not systematic approach. So I am pretty happy with mu current result :)\n![capturebest](https://i.ibb.co/sJV9TH4/Capturebest.png)\n\n", "votes": 1 } ] }, { "id": 936159, "postDate": "2020-07-20T03:06:50.830Z", "content": "

Actually I'm curious, why do you think we see such a difference between pytorch and tf? why pytorch seems to be giving lower scores? Anyone can give some hints?

", "rawMarkdown": "Actually I'm curious, why do you think we see such a difference between pytorch and tf? why pytorch seems to be giving lower scores? Anyone can give some hints?", "votes": 1, "replies": [ { "id": 936362, "postDate": "2020-07-20T07:02:40.950Z", "content": "

Fell in love with Pytorch by way of fastai last year - but did not fully commit waiting for the July publication of new Fastai book by Jermery and crew and version 2.

\n\n

But tensorflow 2 has made me reluctant to start reading the book even though I have purchased it. It was a real pain to use my dual GPU's on Fastai. It seems even more painful to run TPU's.

\n\n

The tensorflow 2 combined with TPU for problems that benefit from large image size seems like this years winner. For sure on Kaggle kernels where time is not your friend and even on my local machines were I can run a model for a couple of days.

\n\n

All this was facilitated by the efforts of several folks and Chris in particular.

\n\n

Would be interested to see if any Pytorch users can pull me back:)

", "rawMarkdown": "Fell in love with Pytorch by way of fastai last year - but did not fully commit waiting for the July publication of new Fastai book by Jermery and crew and version 2.\n\nBut tensorflow 2 has made me reluctant to start reading the book even though I have purchased it. It was a real pain to use my dual GPU's on Fastai. It seems even more painful to run TPU's.\n\nThe tensorflow 2 combined with TPU for problems that benefit from large image size seems like this years winner. For sure on Kaggle kernels where time is not your friend and even on my local machines were I can run a model for a couple of days.\n\nAll this was facilitated by the efforts of several folks and [Chris](https://www.kaggle.com/cdeotte) in particular. \n\nWould be interested to see if any Pytorch users can pull me back:)" }, { "id": 938294, "postDate": "2020-07-21T12:43:16.210Z", "content": "

So far, I can't only acknowledge what you're saying. Tensorflow and TPU are the real winners. In Jigsaw competition already, training XLM-Roberta with PyTorch TPU was impossible on Kaggle. We had to use Tensorflow. It is the only way on Kaggle to iterate really fast.

", "rawMarkdown": "So far, I can't only acknowledge what you're saying. Tensorflow and TPU are the real winners. In Jigsaw competition already, training XLM-Roberta with PyTorch TPU was impossible on Kaggle. We had to use Tensorflow. It is the only way on Kaggle to iterate really fast.", "votes": -1 }, { "id": 939640, "postDate": "2020-07-22T10:53:18.320Z", "content": "
\n

DHZM asked
\n Actually I'm curious, why do you think we see such a difference between pytorch and tf? why pytorch seems to be giving lower scores? Anyone can give some hints?

\n
\n

IMHO, It somewhat feels weird to me to see a discussion on the achieved score by a single framework. We already have this thread for this discussion in general. However, for now, TF has some advantages over PyTorch while using TPU.

", "rawMarkdown": "> **DHZM** asked\nActually I'm curious, why do you think we see such a difference between pytorch and tf? why pytorch seems to be giving lower scores? Anyone can give some hints?\n\nIMHO, It somewhat feels weird to me to see a discussion on the achieved score by a single framework. We already have [this thread](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/156027) for this discussion in general. However, for now, `TF` has some advantages over `PyTorch` while using TPU. " } ] }, { "id": 935915, "postDate": "2020-07-19T19:16:45.863Z", "content": "

FYI: https://www.kaggle.com/c/liverpool-ion-switching/discussion/145256\nIn the past liverpool competition, TF performs better than PyTorch. Psi found it might be the initialization issue in PyTorch.

", "rawMarkdown": "FYI: https://www.kaggle.com/c/liverpool-ion-switching/discussion/145256\nIn the past liverpool competition, TF performs better than PyTorch. Psi found it might be the initialization issue in PyTorch. ", "votes": 1, "replies": [ { "id": 935989, "postDate": "2020-07-19T21:42:57.867Z", "content": "

Isn’t everyone starting from pretrained models? There is no random initialization with transfer learning.

", "rawMarkdown": "Isn’t everyone starting from pretrained models? There is no random initialization with transfer learning." }, { "id": 935997, "postDate": "2020-07-19T21:56:00.937Z", "content": "

Yes the image feature extraction layers are all pretrained. But there might also be some classification layers or meta data layers, these are random initialized. I don't know if it is important, just an idea.

", "rawMarkdown": "Yes the image feature extraction layers are all pretrained. But there might also be some classification layers or meta data layers, these are random initialized. I don't know if it is important, just an idea." } ] }, { "id": 935754, "postDate": "2020-07-19T15:55:05.380Z", "content": "

Thank you guys !

\n\n

You give me motivation to do experiments with 512x512. \n@optimo not that much TTA , just one horizontal flipping.

\n\n

I have save my best checkpoints fortunately. I will give it a try.

", "rawMarkdown": "Thank you guys !\n\nYou give me motivation to do experiments with 512x512. \n@optimo not that much TTA , just one horizontal flipping. \n\nI have save my best checkpoints fortunately. I will give it a try. ", "votes": 1, "replies": [ { "id": 935780, "postDate": "2020-07-19T16:18:47.480Z", "content": "

Would be interested to know how much you were able to gain with TTA!

", "rawMarkdown": "Would be interested to know how much you were able to gain with TTA!", "votes": 1 }, { "id": 935787, "postDate": "2020-07-19T16:26:57.977Z", "content": "

Sure ! I will publish it.

\n\n

I have no submission left for today :)

", "rawMarkdown": "Sure ! I will publish it.\n\nI have no submission left for today :)", "votes": 1 } ] }, { "id": 935558, "postDate": "2020-07-19T13:46:29.037Z", "content": "

Thanks for voicing out what I wanted to ask as well. Seems like more and more competitions are geared towards TF + TPU \nBy the way, is your 0.944 EfficientnetB5 your LB or your local CV? I'm using @cdeotte csv fold too but the validation score barely hit 0.9. I think it was discussed in his notebook too

", "rawMarkdown": "Thanks for voicing out what I wanted to ask as well. Seems like more and more competitions are geared towards TF + TPU \nBy the way, is your 0.944 EfficientnetB5 your LB or your local CV? I'm using @cdeotte csv fold too but the validation score barely hit 0.9. I think it was discussed in his notebook too", "votes": 1, "replies": [ { "id": 935591, "postDate": "2020-07-19T14:04:32.377Z", "content": "

It's the LB score..

\n\n

I got +0.92 on CV

", "rawMarkdown": "It's the LB score..\n\nI got +0.92 on CV", "votes": 1 }, { "id": 936798, "postDate": "2020-07-20T14:31:30.633Z", "content": "

Guys, you have mentioned @cdeotte csv and jpeg files. I have found only tfrecords. Can you please provide link to csv and jpeg files

", "rawMarkdown": "Guys, you have mentioned @cdeotte csv and jpeg files. I have found only tfrecords. Can you please provide link to csv and jpeg files " }, { "id": 936803, "postDate": "2020-07-20T14:36:27.300Z", "content": "

Just goto Data, search cdeotte you will see all the stuff he has posted. You can even further filter by instead searching for cdeotte jpeg

", "rawMarkdown": "Just goto Data, search cdeotte you will see all the stuff he has posted. You can even further filter by instead searching for cdeotte jpeg" }, { "id": 936867, "postDate": "2020-07-20T15:04:25.850Z", "content": "

@vladimirsydor All links are here

", "rawMarkdown": "@vladimirsydor All links are [here][1]\n\n[1]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/164092", "votes": 1 } ] }, { "id": 944090, "postDate": "2020-07-24T20:11:50.683Z", "content": "

Guys what's the best CV (Not LB) you got using image only data . For starter mine is 0.9304\n[UPDATE] Mine is 0.938 now.

", "rawMarkdown": "Guys what's the best CV (Not LB) you got using image only data . For starter mine is 0.9304\n[UPDATE] Mine is 0.938 now.", "votes": 2, "replies": [ { "id": 944151, "postDate": "2020-07-24T21:50:12.513Z", "content": "

Ours is 0.9342 single model with only images\nWhat's the best cv with table only data?\nOurs is 0.832

", "rawMarkdown": "Ours is 0.9342 single model with only images\nWhat's the best cv with table only data?\nOurs is 0.832", "votes": 1 }, { "id": 944274, "postDate": "2020-07-25T01:47:48.153Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 944836, "postDate": "2020-07-25T11:32:06.467Z", "content": "

@tanulsingh077 only metadata model: CV .8582, LB .8343 without ext data, NN model.

", "rawMarkdown": "@tanulsingh077 only metadata model: CV .8582, LB .8343 without ext data, NN model.", "votes": 1 }, { "id": 944864, "postDate": "2020-07-25T11:56:10.020Z", "content": "

That's great , @shivamcyborg do you mind telling me number of features used ?

", "rawMarkdown": "That's great , @shivamcyborg do you mind telling me number of features used ?" }, { "id": 945362, "postDate": "2020-07-25T18:54:17.913Z", "content": "

@tanulsingh077 i am using 8 features.

", "rawMarkdown": "@tanulsingh077 i am using 8 features.", "votes": 1 }, { "id": 946975, "postDate": "2020-07-27T01:51:58.917Z", "content": "

@rohitsingh9990 CV 938 is a great CV score. Are you using single model (5 fold), triple stratified, and only 2020 data in validation?

", "rawMarkdown": "@rohitsingh9990 CV 938 is a great CV score. Are you using single model (5 fold), triple stratified, and only 2020 data in validation?", "votes": 3 }, { "id": 946976, "postDate": "2020-07-27T01:52:12.990Z", "content": "

CV is ~0.92 and LB is ~0.92 as well. The CV and LB scores seem quite consistent for me...

", "rawMarkdown": "CV is ~0.92 and LB is ~0.92 as well. The CV and LB scores seem quite consistent for me..." }, { "id": 947020, "postDate": "2020-07-27T02:34:32.227Z", "rawMarkdown": "", "isDeleted": true }, { "id": 947075, "postDate": "2020-07-27T04:15:14.023Z", "content": "

@cdeotte i am using single model (5 fold) and your triple stratified data, (2020 + 2018) data for training and 2020 data for validation. 0.938 is the Cv for my best fold, average CV for 5 folds is around 0.930.

", "rawMarkdown": "@cdeotte i am using single model (5 fold) and your triple stratified data, (2020 + 2018) data for training and 2020 data for validation. 0.938 is the Cv for my best fold, average CV for 5 folds is around 0.930.", "votes": 2 } ] }, { "id": 939292, "postDate": "2020-07-22T06:29:09.143Z", "content": "

With the same config and 512x512, I got 0.9475 for just the first fold : validation AUC : +0.93.

\n

Things look promising for the 5 folds running :)

", "rawMarkdown": "With the same config and 512x512, I got 0.9475 for just the first fold : validation AUC : +0.93.\n\nThings look promising for the 5 folds running :) ", "votes": 2 }, { "id": 936991, "postDate": "2020-07-20T16:35:06.037Z", "content": "

EfficientNet b4 on 384x384 jpgs by Chris.\nNo external data,no tta.\nLB - 9216

", "rawMarkdown": "EfficientNet b4 on 384x384 jpgs by Chris.\nNo external data,no tta.\nLB - 9216 ", "votes": 2 }, { "id": 936191, "postDate": "2020-07-20T04:00:35.203Z", "content": "

PyTorch
\nLB : 0.921
\nModel : Efficientnet-b0
\nImage size : 256x256
\nAugmentations : Uniform Augment(paper)
\nNo Meta data,
\nNo External data,
\nTTA
\n--> notebook

\n

when I sized up images to 512 with b3, I got 0.935LB.

", "rawMarkdown": "PyTorch\nLB : 0.921\nModel : Efficientnet-b0\nImage size : 256x256\nAugmentations : Uniform Augment([paper](https://arxiv.org/abs/2003.14348))\nNo Meta data,\nNo External data,\nTTA\n--> [notebook](https://www.kaggle.com/ttt2209181/melanoma-classification-with-uniform-augment-x256/notebook)\n\nwhen I sized up images to 512 with b3, I got 0.935LB.", "votes": 2 }, { "id": 935705, "postDate": "2020-07-19T15:22:06.613Z", "content": "

0.944, 5-fold EfficientNet-B4, 512x512, trained on single 1080Ti

", "rawMarkdown": "0.944, 5-fold EfficientNet-B4, 512x512, trained on single 1080Ti ", "votes": 2, "replies": [ { "id": 936070, "postDate": "2020-07-20T00:19:54.380Z", "content": "

how much time was needed to train that on local GPU?

", "rawMarkdown": "how much time was needed to train that on local GPU?" }, { "id": 936091, "postDate": "2020-07-20T01:05:05.697Z", "content": "

@vaillant have u used external data?

", "rawMarkdown": "@vaillant have u used external data?" }, { "id": 936122, "postDate": "2020-07-20T02:25:11.200Z", "content": "

wow single 1080ti? What is your batch size? must be 24 or less?

", "rawMarkdown": "wow single 1080ti? What is your batch size? must be 24 or less?" }, { "id": 936132, "postDate": "2020-07-20T02:35:12.777Z", "content": "

@brianfeeny what is your batch size on Kaggle? I am doing efficientnet-b1 now and batch_size is 64 so for b4 I expect something like 24 or less

", "rawMarkdown": "@brianfeeny what is your batch size on Kaggle? I am doing efficientnet-b1 now and batch_size is 64 so for b4 I expect something like 24 or less" }, { "id": 936138, "postDate": "2020-07-20T02:41:15.263Z", "content": "

I am using 4 GPU's and batch size is about 34 per GPU so 136 @ B2.

", "rawMarkdown": "I am using 4 GPU's and batch size is about 34 per GPU so 136 @ B2." }, { "id": 936168, "postDate": "2020-07-20T03:23:04.863Z", "content": "

@brianfeeny does it mean you have batch size 136 or you are calculating 4 batches of size 34 at once?

", "rawMarkdown": "@brianfeeny does it mean you have batch size 136 or you are calculating 4 batches of size 34 at once?" }, { "id": 936401, "postDate": "2020-07-20T07:30:40.793Z", "content": "

I set batch size to 136 which puts 34 on each GPU.

", "rawMarkdown": "I set batch size to 136 which puts 34 on each GPU." }, { "id": 936404, "postDate": "2020-07-20T07:32:24.590Z", "content": "

@brianfeeny how does it work? can you send me link to kernel or article about it?

", "rawMarkdown": "@brianfeeny how does it work? can you send me link to kernel or article about it?" }, { "id": 936484, "postDate": "2020-07-20T08:52:36.083Z", "content": "

How does what work? Batch Size? I simply inquired about the batch size you were using since you were using a single GPU with 512x512 images on a B4 model. You asked me what size I use, and its 136, but I am using multi-gpu so its really putting 34 on each GPU behind the scenes.

", "rawMarkdown": "How does what work? Batch Size? I simply inquired about the batch size you were using since you were using a single GPU with 512x512 images on a B4 model. You asked me what size I use, and its 136, but I am using multi-gpu so its really putting 34 on each GPU behind the scenes." }, { "id": 936530, "postDate": "2020-07-20T09:48:04.597Z", "content": "

@brianfeeny I mean do you need to implement this multi GPU training in some special way in pytorch, just like we need special way for multicore TPU? I put batch_size in my code to initialize dataloader then in the iteration I take batch and I call model on my GPU, how it works when there are multiple GPUs?

", "rawMarkdown": "@brianfeeny I mean do you need to implement this multi GPU training in some special way in pytorch, just like we need special way for multicore TPU? I put batch_size in my code to initialize dataloader then in the iteration I take batch and I call model on my GPU, how it works when there are multiple GPUs?" }, { "id": 936566, "postDate": "2020-07-20T10:17:20.817Z", "content": "

There are multiple ways to use multiple GPU’s in pytorch. The easiest is called DataParallel, and it only requires you add one line to your code.

\n\n

Instead of:

\n\n

model = (...)\nmodel.to(device)

\n\n

Do:

\n\n

model = (...)\nmodel = nn.DataParallel(model)\nmodel.to(device)

", "rawMarkdown": "There are multiple ways to use multiple GPU’s in pytorch. The easiest is called DataParallel, and it only requires you add one line to your code.\n\nInstead of:\n\nmodel = (...)\nmodel.to(device)\n\nDo:\n\nmodel = (...)\nmodel = nn.DataParallel(model)\nmodel.to(device)", "votes": 1 }, { "id": 936584, "postDate": "2020-07-20T10:31:33.730Z", "content": "

Thank you.

", "rawMarkdown": "Thank you." } ] }, { "id": 935638, "postDate": "2020-07-19T14:33:21.390Z", "content": "

My single model score is only 0.9478.\nNo more feature engineering is built.

", "rawMarkdown": "My single model score is only 0.9478.\nNo more feature engineering is built." }, { "id": 938290, "postDate": "2020-07-21T12:40:40.503Z", "content": "

EfficientNetB6, Image size 384x384, 5-fold stratified with external data
\nUsing metafeatures: CV: 0.9217 LB: 0.9409
\nWithout metafeatures: CV: 0.9202 LB: 0.9355
\nUsing PyTorch

", "rawMarkdown": "EfficientNetB6, Image size 384x384, 5-fold stratified with external data\nUsing metafeatures: CV: 0.9217 LB: 0.9409\nWithout metafeatures: CV: 0.9202 LB: 0.9355\nUsing PyTorch", "votes": 1, "replies": [ { "id": 938693, "postDate": "2020-07-21T17:25:25.490Z", "content": "

What was the training time ? Also how many epochs per fold ?

", "rawMarkdown": "What was the training time ? Also how many epochs per fold ?" }, { "id": 938826, "postDate": "2020-07-21T19:25:23.970Z", "content": "

3 hours / fold , 15 epochs \nwe train two folds/kernel on kaggle due to time limitations

", "rawMarkdown": "3 hours / fold , 15 epochs \nwe train two folds/kernel on kaggle due to time limitations" } ] }, { "id": 972666, "postDate": "2020-08-16T18:11:37.903Z", "content": "

EfficientNet-B4, 256x256, 5-Fold, LB 0.9316

\n

Btw what do you mean by \"[…] 15 TTA for each fold.\"
\n15 Augmentations blended together with the original image? That seems a lot to me.

", "rawMarkdown": "EfficientNet-B4, 256x256, 5-Fold, LB 0.9316\n\nBtw what do you mean by \"[...] 15 TTA for each fold.\"\n15 Augmentations blended together with the original image? That seems a lot to me." }, { "id": 956304, "postDate": "2020-08-03T11:56:32.150Z", "content": "

effb5 on 384x384 got 0.9432 with TTA 11 ( increasing TTA to 23 got 0.9422)\nTrying now on 512x512

", "rawMarkdown": "effb5 on 384x384 got 0.9432 with TTA 11 ( increasing TTA to 23 got 0.9422)\nTrying now on 512x512 " }, { "id": 946636, "postDate": "2020-07-26T17:34:17.070Z", "content": "

EfficientNet B2 384x384\nSingle Fold\nCV 92.9, LB 93.5 using TTA

", "rawMarkdown": "EfficientNet B2 384x384\nSingle Fold\nCV 92.9, LB 93.5 using TTA" }, { "id": 945213, "postDate": "2020-07-25T16:52:00.303Z", "content": "

Something interesting i've experienced: my CV score between image sizes stays the same, around 0.9375 single fold.\nHowever my lb is different depending on the img size: 300x300 0.9266, 380x380 0.9310, 512x512 0.9379. These were all trained with the same setup (Effnetb4) and fold.

", "rawMarkdown": "Something interesting i've experienced: my CV score between image sizes stays the same, around 0.9375 single fold.\nHowever my lb is different depending on the img size: 300x300 0.9266, 380x380 0.9310, 512x512 0.9379. These were all trained with the same setup (Effnetb4) and fold.", "replies": [ { "id": 945444, "postDate": "2020-07-25T20:06:50.637Z", "content": "

@yannmajewski did you train it with one size and then feed it the other sizes, or did you train it each time with each of these sizes?

", "rawMarkdown": "@yannmajewski did you train it with one size and then feed it the other sizes, or did you train it each time with each of these sizes?" }, { "id": 945598, "postDate": "2020-07-26T02:05:09.597Z", "content": "

@brianfeeny each time its a different experiment with a new model!

", "rawMarkdown": "@brianfeeny each time its a different experiment with a new model!" } ] }, { "id": 944908, "postDate": "2020-07-25T12:51:07.360Z", "content": "

Using the same seed, I can usually get pretty consistent results with the same run. For this dataset, though, I'm getting as much as +/-0.005 on the same fold with the same setup. (This is PyTorch, btw)

", "rawMarkdown": "Using the same seed, I can usually get pretty consistent results with the same run. For this dataset, though, I'm getting as much as +/-0.005 on the same fold with the same setup. (This is PyTorch, btw)", "replies": [ { "id": 945166, "postDate": "2020-07-25T15:57:09.603Z", "content": "

pytorch allows you to have reproducible results, even with GPU computation, so if you fix every random seed you should have exactly the same results, you probably have some free seed somewhere that creates this.

", "rawMarkdown": "pytorch allows you to have reproducible results, even with GPU computation, so if you fix every random seed you should have exactly the same results, you probably have some free seed somewhere that creates this. " } ] }, { "id": 942614, "postDate": "2020-07-23T21:45:14.667Z", "content": "

It's amazing seeing the huge models and image sizes in this thread. I think I need to fit a nitrous bottle to my computer to keep up with you guys 🏎

", "rawMarkdown": "It's amazing seeing the huge models and image sizes in this thread. I think I need to fit a nitrous bottle to my computer to keep up with you guys 🏎 " }, { "id": 936080, "postDate": "2020-07-20T00:42:36.767Z", "content": "

Thanks for the useful info and discussion that it's started. Could you please share information on which augmentations you used in PyTorch and how you selected them? Thanks!

", "rawMarkdown": "Thanks for the useful info and discussion that it's started. Could you please share information on which augmentations you used in PyTorch and how you selected them? Thanks!" }, { "id": 936072, "postDate": "2020-07-20T00:21:58.263Z", "content": "

finally got some stability\n256x256, efficientnet-b0, single fold\ntrained on local GPU\nLB 0.9176 \nlocal 0.9166

", "rawMarkdown": "finally got some stability\n256x256, efficientnet-b0, single fold\ntrained on local GPU\nLB 0.9176 \nlocal 0.9166" }, { "id": 935927, "postDate": "2020-07-19T19:30:10.830Z", "content": "

LB : 0.944 Effb4\nImg size: 512x512\nWith external data, no TTA :)

", "rawMarkdown": "LB : 0.944 Effb4\nImg size: 512x512\nWith external data, no TTA :)", "replies": [ { "id": 936093, "postDate": "2020-07-20T01:09:08.050Z", "content": "

@rohan1602 have u used only image features or meta-features as well?

", "rawMarkdown": "@rohan1602 have u used only image features or meta-features as well?" }, { "id": 936199, "postDate": "2020-07-20T04:17:51.323Z", "rawMarkdown": "", "isDeleted": true }, { "id": 936306, "postDate": "2020-07-20T06:23:25.637Z", "content": "

@rohitsingh9990 The model has both image and meta features as input.\n@synked CV: 93.9..I 'm still working on getting a stable CV.

", "rawMarkdown": "@rohitsingh9990 The model has both image and meta features as input.\n@synked CV: 93.9..I 'm still working on getting a stable CV.", "votes": 1 } ] }, { "id": 935888, "postDate": "2020-07-19T18:44:20.413Z", "content": "

192x192 with no meta data on triple stratified tfrecords, LB score 9.449, cv 0.937

", "rawMarkdown": "192x192 with no meta data on triple stratified tfrecords, LB score 9.449, cv 0.937", "replies": [ { "id": 936124, "postDate": "2020-07-20T02:28:14.703Z", "content": "

what network arch?

", "rawMarkdown": "what network arch?" }, { "id": 936197, "postDate": "2020-07-20T04:10:02.117Z", "content": "

Seresnet50

", "rawMarkdown": "Seresnet50" }, { "id": 943369, "postDate": "2020-07-24T10:14:56.607Z", "content": "

That sounds really good.

", "rawMarkdown": "That sounds really good." } ] }, { "id": 935839, "postDate": "2020-07-19T17:37:16.713Z", "content": "

20X TTA!! Good lord. I have rarely seen such high TTA rounds. Thanks for the info though, could be an interesting experiment for me.

", "rawMarkdown": "20X TTA!! Good lord. I have rarely seen such high TTA rounds. Thanks for the info though, could be an interesting experiment for me." }, { "id": 972704, "postDate": "2020-08-16T18:40:33.013Z", "rawMarkdown": "", "isDeleted": true }, { "id": 972177, "postDate": "2020-08-16T09:55:34.827Z", "rawMarkdown": "", "votes": -2, "isDeleted": true }, { "id": 964601, "postDate": "2020-08-10T03:20:30.977Z", "rawMarkdown": "", "isDeleted": true }, { "id": 935837, "postDate": "2020-07-19T17:36:22.870Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 939957, "author_name": "yuval reina", "author_url": "", "post_date": "2020-07-22T15:36:12.797000", "content": "

I got >0.95 with a single model which is of similar size as your model and after adding some sauce to it (model wise, no LB probing) I got 0.96 from a single model. But you should not take every LB result as is, the test set is small and very un-balanced with very low positive examples. a high Public LB score can turn into a low Private LB. If I judge by my CV std - Shakeup is Coming

", "votes": 14, "replies": [ { "id": 939969, "author_name": "Innat", "author_url": "", "post_date": "2020-07-22T15:48:03.497000", "content": "

0.96 by single model! 💥
\nThat sauce must be damn tasty!

", "votes": 1, "replies": [] }, { "id": 940020, "author_name": "Optimo", "author_url": "", "post_date": "2020-07-22T16:17:58.943000", "content": "

Thanks for sharing!

\n

Have you been able to reproduce your single score >0.96? or it's been a one shot?

", "votes": 1, "replies": [] }, { "id": 940130, "author_name": "Serigne ", "author_url": "", "post_date": "2020-07-22T17:27:10.670000", "content": "

Thanks @yuval6967 for the feedback!\nI will definitely trust my CV

", "votes": 1, "replies": [] }, { "id": 940331, "author_name": "Serigne ", "author_url": "", "post_date": "2020-07-22T20:46:46.500000", "content": "

@optimo I don't know about @yuval6967 but I'm having deterministic results with Pytorch.

\n\n

It's impossible with Tensorflow

", "votes": 1, "replies": [] }, { "id": 940336, "author_name": "Optimo", "author_url": "", "post_date": "2020-07-22T20:56:47.463000", "content": "

Well @serigne yes I meant reproduce the same result with a different random seed, independently of being able to fix everything. Would this model produce above 0.96 with any random seed?

", "votes": 0, "replies": [] }, { "id": 940339, "author_name": "Serigne ", "author_url": "", "post_date": "2020-07-22T21:04:53.380000", "content": "

Random seed is a feature as another. Of course you may have different results by changing it.

\n\n

But the difference should be small to make sure the model is stable.

", "votes": 0, "replies": [] }, { "id": 941857, "author_name": "yuval reina", "author_url": "", "post_date": "2020-07-23T13:02:27.640000", "content": "

@optimo yes I was able to get a similar results with a slightly different model and different training parameters + different seed. As the LB is very noisy, the result won't be reproduced with any random seed, my CV std is ~0.004 (between folds). (I stopped aiming for the highest Public LB long ago as I don't really believe in it)

", "votes": 2, "replies": [] }, { "id": 941972, "author_name": "Optimo", "author_url": "", "post_date": "2020-07-23T14:22:40.843000", "content": "

@yuval6967 thanks! I agree, it takes some wisdom to refrain from climbing Public LB... but Private LB will bring some nice reward! Good luck!

", "votes": 1, "replies": [] } ] }, { "id": 946965, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-27T01:41:33.627000", "content": "

Note: Everyone is reporting their CV here. We should also report how our CV is setup because without knowing what data people are adding to their validation folds and whether people are using triple stratified or not, we cannot compare CVs.

\n\n

For example, if you use external data, don't stratify, don't remove duplicate images, and allow external data in your validation fold, it is easy to get 5 KFold CV AUC 0.98+

", "votes": 6, "replies": [ { "id": 947368, "author_name": "Serigne ", "author_url": "", "post_date": "2020-07-27T08:07:53.797000", "content": "

Using external data in validation doesn't make sense indeed.

\n\n

What I did, was to create firstly K-fold from your Triplet Stratified CSV of 2020 data. Then I added a part of external data on the training part for each fold.
\nhowever I still noticed huge gap between folds or between some folds and their corresponding LB submission. That why I don't rely very much on regular CV and try instead Snapshot training and multiple checkpoints averaging.

", "votes": 1, "replies": [] }, { "id": 947400, "author_name": "Dracarys", "author_url": "", "post_date": "2020-07-27T08:43:18.757000", "content": "

@serigne you are right about variations in CV of each fold, in my case my per fold CV varies from 0.92-0.94

", "votes": 1, "replies": [] }, { "id": 948288, "author_name": "datasaurus", "author_url": "", "post_date": "2020-07-27T19:27:58.290000", "content": "

I guess it's also worth mentioning that taking the mean of the k AUC scores from each fold is not equal to calculating the global AUC from the out of fold predictions

", "votes": 1, "replies": [] }, { "id": 948290, "author_name": "Jacek Poplawski", "author_url": "", "post_date": "2020-07-27T19:30:03.900000", "content": "

spoiler alert: using external data in validation is one of the largest failures which people do in this competition

", "votes": 2, "replies": [] }, { "id": 948354, "author_name": "Signal", "author_url": "", "post_date": "2020-07-27T21:00:25.407000", "content": "

@jacekpoplawski I don't know about that, but certainly avoiding it is easy.

\n\n

Say your train/non-external data goes up to index 32000. Then you concatenate external data 1, external data 2, external data 3......etc. Now you do your splits/folds (using Scikit or whatever). You now have list of train_indexes and val_indexes. Simply do the following to the val_indexes at the top of each fold:

\n\n

```\nval_indexes = val_indexes[val_indexes <= 32000]

\n\n

```\nNote, I am assuming here that val_indexes is a Numpy array (which is what Scikit produces).

", "votes": 0, "replies": [] } ] }, { "id": 974360, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-17T23:42:35.297000", "content": "

Could not tune effnet b5, my best is b4 with LB 0.9474, and a bit over 95 with a blend.

\n

I did not blend with any public kernels, couldn't resolve to it. There is a limit to what I am ready to do to get Kaggle points, LOL.

\n

I regret I forgot to try two ideas I had but5 overall it was a great learning experience. Maybe 10 years form now I'll be competitive in computer vision ;) Maybe I should just buy code, I see some gave pointers to relevant online stores…

", "votes": 3, "replies": [ { "id": 974371, "author_name": "Signal", "author_url": "", "post_date": "2020-08-17T23:50:54.800000", "content": "

@cpmpml I can respect that. I did not blend any public kernel either and I am right where you are. Integrity is one of the if not the most important characteristic in Science. If I medal it is in no doubt due to many contributions shared by many people, but those contributions are not going to be scores that I \"blend\".

", "votes": 1, "replies": [] }, { "id": 974380, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-17T23:59:01.187000", "content": "

Yes, we seem to have very similar paths in this comp. Your code snippets helped me a lot, thanks again.

", "votes": 1, "replies": [] } ] }, { "id": 972452, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-16T15:06:06.943000", "content": "

effnetb4, images only, tta, CV 9440 LB 9452

\n

I started with b5 but CV and LB are similar, a bit lower. Trying to tune for a last run …

\n

I think my final rank will be similar to my current rank.

", "votes": 3, "replies": [ { "id": 972463, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-16T15:10:25.363000", "content": "

Nice job, great CV

", "votes": 1, "replies": [] }, { "id": 972494, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-16T15:28:48.723000", "content": "

Thanks Chris. Great CV and poor LB ;)

", "votes": 0, "replies": [] }, { "id": 972770, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2020-08-16T20:12:30.557000", "content": "

That is a very good CV.

", "votes": 1, "replies": [] } ] }, { "id": 935732, "author_name": "Optimo", "author_url": "", "post_date": "2020-07-19T15:36:38.050000", "content": "

0.9458 effnet-b2, 512x512, 5 fold with 20xTTA, ext data + metadata, local CV 0.925

\n

@serigne have you tried TTA? if you have some heavy augmentation it could work.

", "votes": 3, "replies": [ { "id": 936059, "author_name": "sin", "author_url": "", "post_date": "2020-07-19T23:58:25.247000", "content": "

which TTAs are you referring to?

", "votes": 0, "replies": [] }, { "id": 936303, "author_name": "Optimo", "author_url": "", "post_date": "2020-07-20T06:20:23.997000", "content": "

Just leaving my training augmentation ON for the test predictions

", "votes": 0, "replies": [] } ] }, { "id": 937048, "author_name": "Tahsin Mostafiz", "author_url": "", "post_date": "2020-07-20T17:31:11.233000", "content": "

0.9424 on LB and 0.9407 on Validation\nSingle Eff B5 Only\nImage Dimension 456x456. Used external data\nPytorch

", "votes": 4, "replies": [ { "id": 938542, "author_name": "Optimo", "author_url": "", "post_date": "2020-07-21T15:44:14.973000", "content": "

May I ask how you tighten the gap between best pytorch model 0.9424 and a score of 95.5 on LB?

\n

A- better single model in tensorflow?
\nB- ensembling of multiple models (no public)?
\nC- ensembling with public notebooks?
\nD- A mix of all this

\n

I'm asking this because I'm very worried that everyone is currently overfiting public LB because of lucky public seeds, so A-B -> I'll try hard more C-D -> I stop trying to climb the LB

", "votes": 1, "replies": [] }, { "id": 938601, "author_name": "Idris", "author_url": "", "post_date": "2020-07-21T16:15:26.823000", "content": "

@optimo for me, the same model with different seeds score : 0.9531 and 0.9502

", "votes": 1, "replies": [] }, { "id": 938642, "author_name": "Optimo", "author_url": "", "post_date": "2020-07-21T16:41:49.030000", "content": "

Well you seem to be in another planet at the moment^^ congrats!

\n

But the same question could apply to you: is your best model scoring \"only\" 0.9531? Then how do you climb up to 0.9644?

\n

I don't want a secret recipe just to be sure that everyone is not stacking themselves to the same notebooks that scores well on public LB. Are you using any public notebook in your ensemble? (I definitely hope that in order to go top 3 at the moment you are not using a public blender with custom weights but that would be a great info!)

\n

But I guess my question is more towards people ranking from 600th to 50th, as I'm definitely sure that the shake up will be more severe in these positions.

\n

Thanks!

", "votes": 1, "replies": [] }, { "id": 939437, "author_name": "Tahsin Mostafiz", "author_url": "", "post_date": "2020-07-22T08:45:11.973000", "content": "

@optimo I have used this dataset. It comes with multiple folds and I used them. I have been thinking of switching to Triple Stratified Leak-Free KFold CV because it seems more logical. And all the credits for achieving 95.5+ on public LB go to my teammate @ipythonx. He is really good at ensembling. But we are also very much concerned about overfitting the public LB.

", "votes": 0, "replies": [] }, { "id": 939498, "author_name": "Optimo", "author_url": "", "post_date": "2020-07-22T09:12:29.817000", "content": "

thanks @tahsin it might turn out to be an ensemble war then! ^^

", "votes": 1, "replies": [] }, { "id": 939505, "author_name": "Tahsin Mostafiz", "author_url": "", "post_date": "2020-07-22T09:20:20.513000", "content": "

I think so too. Try to get as much as possible 95+ scores with single models. Then ensemble and pray.

", "votes": 0, "replies": [] }, { "id": 943341, "author_name": "VinayVikram", "author_url": "", "post_date": "2020-07-24T09:56:06.380000", "content": "

@tahsin I am also getting LB score around 95.8 but very much concerned about overfitting,, Like discuss what techniques are you planning to avoid overfitting.

", "votes": 0, "replies": [] }, { "id": 943359, "author_name": "Optimo", "author_url": "", "post_date": "2020-07-24T10:05:56.057000", "content": "

I think he mentioned praying! 🙏 ^^

", "votes": 3, "replies": [] } ] }, { "id": 936694, "author_name": "Serigne ", "author_url": "", "post_date": "2020-07-20T13:01:41.417000", "content": "

Thanks @optimo

\n\n

I have run 15 TTA for one fold ..

\n\n

it improved the score from 0.9293 to 0.9387.

\n\n

I will update the overall score once it's done for all 5 folds.

", "votes": 4, "replies": [ { "id": 937829, "author_name": "shiba", "author_url": "", "post_date": "2020-07-21T07:22:16.970000", "content": "

15TTA.......

", "votes": 0, "replies": [] }, { "id": 937962, "author_name": "Serigne ", "author_url": "", "post_date": "2020-07-21T09:13:49.343000", "content": "

It tooks me about 40 minutes to run for each fold on GPU , but the score improvement worth it :)

\n

I will like parallelize it on 8 cores TPU for the next expeiments.

", "votes": 0, "replies": [] }, { "id": 938546, "author_name": "Optimo", "author_url": "", "post_date": "2020-07-21T15:45:42.367000", "content": "

so @serigne what did it give you with 5 folds?

", "votes": 0, "replies": [] }, { "id": 938780, "author_name": "Serigne ", "author_url": "", "post_date": "2020-07-21T18:33:42.970000", "content": "

0.9496 on LB

\n

I'm running 512x512 now on B5. I think it will give better score

", "votes": 2, "replies": [] }, { "id": 947181, "author_name": "Dracarys", "author_url": "", "post_date": "2020-07-27T05:49:14.453000", "content": "

@serigne what's your CV for 0.9496 LB?

", "votes": 0, "replies": [] } ] }, { "id": 974507, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-18T01:00:24.247000", "content": "

@serigne congrats on the good finish.

", "votes": 1, "replies": [ { "id": 974568, "author_name": "Serigne ", "author_url": "", "post_date": "2020-08-18T01:34:19.457000", "content": "

Thank you very much @cpmpml

\n

\"Always trust you CV\" as they say
\n\"CaptureCV\"
\nI knew my best public LB was heavily overfitted , So I didn't even try to choose it, despite given 3 choices.

\n

My chosen private scored \"just\" 0.9594 on public LB

\n

Even though I have also a not chosen gold solution (private LB 0.9467), it was not my best model. Chosing it would be just luck and not systematic approach. So I am pretty happy with mu current result :)
\n\"capturebest\"

", "votes": 1, "replies": [] } ] }, { "id": 936159, "author_name": "DHZM", "author_url": "", "post_date": "2020-07-20T03:06:50.830000", "content": "

Actually I'm curious, why do you think we see such a difference between pytorch and tf? why pytorch seems to be giving lower scores? Anyone can give some hints?

", "votes": 1, "replies": [ { "id": 936362, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2020-07-20T07:02:40.950000", "content": "

Fell in love with Pytorch by way of fastai last year - but did not fully commit waiting for the July publication of new Fastai book by Jermery and crew and version 2.

\n\n

But tensorflow 2 has made me reluctant to start reading the book even though I have purchased it. It was a real pain to use my dual GPU's on Fastai. It seems even more painful to run TPU's.

\n\n

The tensorflow 2 combined with TPU for problems that benefit from large image size seems like this years winner. For sure on Kaggle kernels where time is not your friend and even on my local machines were I can run a model for a couple of days.

\n\n

All this was facilitated by the efforts of several folks and Chris in particular.

\n\n

Would be interested to see if any Pytorch users can pull me back:)

", "votes": 0, "replies": [] }, { "id": 938294, "author_name": "PAB97", "author_url": "", "post_date": "2020-07-21T12:43:16.210000", "content": "

So far, I can't only acknowledge what you're saying. Tensorflow and TPU are the real winners. In Jigsaw competition already, training XLM-Roberta with PyTorch TPU was impossible on Kaggle. We had to use Tensorflow. It is the only way on Kaggle to iterate really fast.

", "votes": -1, "replies": [] }, { "id": 939640, "author_name": "Innat", "author_url": "", "post_date": "2020-07-22T10:53:18.320000", "content": "
\n

DHZM asked
\n Actually I'm curious, why do you think we see such a difference between pytorch and tf? why pytorch seems to be giving lower scores? Anyone can give some hints?

\n
\n

IMHO, It somewhat feels weird to me to see a discussion on the achieved score by a single framework. We already have this thread for this discussion in general. However, for now, TF has some advantages over PyTorch while using TPU.

", "votes": 0, "replies": [] } ] }, { "id": 935915, "author_name": "Waylon Wu", "author_url": "", "post_date": "2020-07-19T19:16:45.863000", "content": "

FYI: https://www.kaggle.com/c/liverpool-ion-switching/discussion/145256\nIn the past liverpool competition, TF performs better than PyTorch. Psi found it might be the initialization issue in PyTorch.

", "votes": 1, "replies": [ { "id": 935989, "author_name": "Optimo", "author_url": "", "post_date": "2020-07-19T21:42:57.867000", "content": "

Isn’t everyone starting from pretrained models? There is no random initialization with transfer learning.

", "votes": 0, "replies": [] }, { "id": 935997, "author_name": "Waylon Wu", "author_url": "", "post_date": "2020-07-19T21:56:00.937000", "content": "

Yes the image feature extraction layers are all pretrained. But there might also be some classification layers or meta data layers, these are random initialized. I don't know if it is important, just an idea.

", "votes": 0, "replies": [] } ] }, { "id": 935754, "author_name": "Serigne ", "author_url": "", "post_date": "2020-07-19T15:55:05.380000", "content": "

Thank you guys !

\n\n

You give me motivation to do experiments with 512x512. \n@optimo not that much TTA , just one horizontal flipping.

\n\n

I have save my best checkpoints fortunately. I will give it a try.

", "votes": 1, "replies": [ { "id": 935780, "author_name": "Optimo", "author_url": "", "post_date": "2020-07-19T16:18:47.480000", "content": "

Would be interested to know how much you were able to gain with TTA!

", "votes": 1, "replies": [] }, { "id": 935787, "author_name": "Serigne ", "author_url": "", "post_date": "2020-07-19T16:26:57.977000", "content": "

Sure ! I will publish it.

\n\n

I have no submission left for today :)

", "votes": 1, "replies": [] } ] }, { "id": 935558, "author_name": "Tung", "author_url": "", "post_date": "2020-07-19T13:46:29.037000", "content": "

Thanks for voicing out what I wanted to ask as well. Seems like more and more competitions are geared towards TF + TPU \nBy the way, is your 0.944 EfficientnetB5 your LB or your local CV? I'm using @cdeotte csv fold too but the validation score barely hit 0.9. I think it was discussed in his notebook too

", "votes": 1, "replies": [ { "id": 935591, "author_name": "Serigne ", "author_url": "", "post_date": "2020-07-19T14:04:32.377000", "content": "

It's the LB score..

\n\n

I got +0.92 on CV

", "votes": 1, "replies": [] }, { "id": 936798, "author_name": "Volodymyr", "author_url": "", "post_date": "2020-07-20T14:31:30.633000", "content": "

Guys, you have mentioned @cdeotte csv and jpeg files. I have found only tfrecords. Can you please provide link to csv and jpeg files

", "votes": 0, "replies": [] }, { "id": 936803, "author_name": "Signal", "author_url": "", "post_date": "2020-07-20T14:36:27.300000", "content": "

Just goto Data, search cdeotte you will see all the stuff he has posted. You can even further filter by instead searching for cdeotte jpeg

", "votes": 0, "replies": [] }, { "id": 936867, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-20T15:04:25.850000", "content": "

@vladimirsydor All links are here

", "votes": 1, "replies": [] } ] }, { "id": 944090, "author_name": "Dracarys", "author_url": "", "post_date": "2020-07-24T20:11:50.683000", "content": "

Guys what's the best CV (Not LB) you got using image only data . For starter mine is 0.9304\n[UPDATE] Mine is 0.938 now.

", "votes": 2, "replies": [ { "id": 944151, "author_name": "Mr_KnowNothing", "author_url": "", "post_date": "2020-07-24T21:50:12.513000", "content": "

Ours is 0.9342 single model with only images\nWhat's the best cv with table only data?\nOurs is 0.832

", "votes": 1, "replies": [] }, { "id": 944274, "author_name": "", "author_url": "", "post_date": "2020-07-25T01:47:48.153000", "content": "", "votes": 1, "replies": [] }, { "id": 944836, "author_name": "Shivam Gupta", "author_url": "", "post_date": "2020-07-25T11:32:06.467000", "content": "

@tanulsingh077 only metadata model: CV .8582, LB .8343 without ext data, NN model.

", "votes": 1, "replies": [] }, { "id": 944864, "author_name": "Mr_KnowNothing", "author_url": "", "post_date": "2020-07-25T11:56:10.020000", "content": "

That's great , @shivamcyborg do you mind telling me number of features used ?

", "votes": 0, "replies": [] }, { "id": 945362, "author_name": "Shivam Gupta", "author_url": "", "post_date": "2020-07-25T18:54:17.913000", "content": "

@tanulsingh077 i am using 8 features.

", "votes": 1, "replies": [] }, { "id": 946975, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-27T01:51:58.917000", "content": "

@rohitsingh9990 CV 938 is a great CV score. Are you using single model (5 fold), triple stratified, and only 2020 data in validation?

", "votes": 3, "replies": [] }, { "id": 946976, "author_name": "X.Chen", "author_url": "", "post_date": "2020-07-27T01:52:12.990000", "content": "

CV is ~0.92 and LB is ~0.92 as well. The CV and LB scores seem quite consistent for me...

", "votes": 0, "replies": [] }, { "id": 947020, "author_name": "", "author_url": "", "post_date": "2020-07-27T02:34:32.227000", "content": "", "votes": 0, "replies": [] }, { "id": 947075, "author_name": "Dracarys", "author_url": "", "post_date": "2020-07-27T04:15:14.023000", "content": "

@cdeotte i am using single model (5 fold) and your triple stratified data, (2020 + 2018) data for training and 2020 data for validation. 0.938 is the Cv for my best fold, average CV for 5 folds is around 0.930.

", "votes": 2, "replies": [] } ] }, { "id": 939292, "author_name": "Serigne ", "author_url": "", "post_date": "2020-07-22T06:29:09.143000", "content": "

With the same config and 512x512, I got 0.9475 for just the first fold : validation AUC : +0.93.

\n

Things look promising for the 5 folds running :)

", "votes": 2, "replies": [] }, { "id": 936991, "author_name": "sayak", "author_url": "", "post_date": "2020-07-20T16:35:06.037000", "content": "

EfficientNet b4 on 384x384 jpgs by Chris.\nNo external data,no tta.\nLB - 9216

", "votes": 2, "replies": [] }, { "id": 936191, "author_name": "yamaru", "author_url": "", "post_date": "2020-07-20T04:00:35.203000", "content": "

PyTorch
\nLB : 0.921
\nModel : Efficientnet-b0
\nImage size : 256x256
\nAugmentations : Uniform Augment(paper)
\nNo Meta data,
\nNo External data,
\nTTA
\n--> notebook

\n

when I sized up images to 512 with b3, I got 0.935LB.

", "votes": 2, "replies": [] }, { "id": 935705, "author_name": "Ian Pan", "author_url": "", "post_date": "2020-07-19T15:22:06.613000", "content": "

0.944, 5-fold EfficientNet-B4, 512x512, trained on single 1080Ti

", "votes": 2, "replies": [ { "id": 936070, "author_name": "Jacek Poplawski", "author_url": "", "post_date": "2020-07-20T00:19:54.380000", "content": "

how much time was needed to train that on local GPU?

", "votes": 0, "replies": [] }, { "id": 936091, "author_name": "Dracarys", "author_url": "", "post_date": "2020-07-20T01:05:05.697000", "content": "

@vaillant have u used external data?

", "votes": 0, "replies": [] }, { "id": 936122, "author_name": "Signal", "author_url": "", "post_date": "2020-07-20T02:25:11.200000", "content": "

wow single 1080ti? What is your batch size? must be 24 or less?

", "votes": 0, "replies": [] }, { "id": 936132, "author_name": "Jacek Poplawski", "author_url": "", "post_date": "2020-07-20T02:35:12.777000", "content": "

@brianfeeny what is your batch size on Kaggle? I am doing efficientnet-b1 now and batch_size is 64 so for b4 I expect something like 24 or less

", "votes": 0, "replies": [] }, { "id": 936138, "author_name": "Signal", "author_url": "", "post_date": "2020-07-20T02:41:15.263000", "content": "

I am using 4 GPU's and batch size is about 34 per GPU so 136 @ B2.

", "votes": 0, "replies": [] }, { "id": 936168, "author_name": "Jacek Poplawski", "author_url": "", "post_date": "2020-07-20T03:23:04.863000", "content": "

@brianfeeny does it mean you have batch size 136 or you are calculating 4 batches of size 34 at once?

", "votes": 0, "replies": [] }, { "id": 936401, "author_name": "Signal", "author_url": "", "post_date": "2020-07-20T07:30:40.793000", "content": "

I set batch size to 136 which puts 34 on each GPU.

", "votes": 0, "replies": [] }, { "id": 936404, "author_name": "Jacek Poplawski", "author_url": "", "post_date": "2020-07-20T07:32:24.590000", "content": "

@brianfeeny how does it work? can you send me link to kernel or article about it?

", "votes": 0, "replies": [] }, { "id": 936484, "author_name": "Signal", "author_url": "", "post_date": "2020-07-20T08:52:36.083000", "content": "

How does what work? Batch Size? I simply inquired about the batch size you were using since you were using a single GPU with 512x512 images on a B4 model. You asked me what size I use, and its 136, but I am using multi-gpu so its really putting 34 on each GPU behind the scenes.

", "votes": 0, "replies": [] }, { "id": 936530, "author_name": "Jacek Poplawski", "author_url": "", "post_date": "2020-07-20T09:48:04.597000", "content": "

@brianfeeny I mean do you need to implement this multi GPU training in some special way in pytorch, just like we need special way for multicore TPU? I put batch_size in my code to initialize dataloader then in the iteration I take batch and I call model on my GPU, how it works when there are multiple GPUs?

", "votes": 0, "replies": [] }, { "id": 936566, "author_name": "Signal", "author_url": "", "post_date": "2020-07-20T10:17:20.817000", "content": "

There are multiple ways to use multiple GPU’s in pytorch. The easiest is called DataParallel, and it only requires you add one line to your code.

\n\n

Instead of:

\n\n

model = (...)\nmodel.to(device)

\n\n

Do:

\n\n

model = (...)\nmodel = nn.DataParallel(model)\nmodel.to(device)

", "votes": 1, "replies": [] }, { "id": 936584, "author_name": "Jacek Poplawski", "author_url": "", "post_date": "2020-07-20T10:31:33.730000", "content": "

Thank you.

", "votes": 0, "replies": [] } ] }, { "id": 935638, "author_name": "Mingjie Wang", "author_url": "", "post_date": "2020-07-19T14:33:21.390000", "content": "

My single model score is only 0.9478.\nNo more feature engineering is built.

", "votes": 0, "replies": [] }, { "id": 938290, "author_name": "PAB97", "author_url": "", "post_date": "2020-07-21T12:40:40.503000", "content": "

EfficientNetB6, Image size 384x384, 5-fold stratified with external data
\nUsing metafeatures: CV: 0.9217 LB: 0.9409
\nWithout metafeatures: CV: 0.9202 LB: 0.9355
\nUsing PyTorch

", "votes": 1, "replies": [ { "id": 938693, "author_name": "Atharva Phatak", "author_url": "", "post_date": "2020-07-21T17:25:25.490000", "content": "

What was the training time ? Also how many epochs per fold ?

", "votes": 0, "replies": [] }, { "id": 938826, "author_name": "Mr_KnowNothing", "author_url": "", "post_date": "2020-07-21T19:25:23.970000", "content": "

3 hours / fold , 15 epochs \nwe train two folds/kernel on kaggle due to time limitations

", "votes": 0, "replies": [] } ] }, { "id": 972666, "author_name": "Ali Abdin", "author_url": "", "post_date": "2020-08-16T18:11:37.903000", "content": "

EfficientNet-B4, 256x256, 5-Fold, LB 0.9316

\n

Btw what do you mean by \"[…] 15 TTA for each fold.\"
\n15 Augmentations blended together with the original image? That seems a lot to me.

", "votes": 0, "replies": [] }, { "id": 956304, "author_name": "Alhasan Abdellatif", "author_url": "", "post_date": "2020-08-03T11:56:32.150000", "content": "

effb5 on 384x384 got 0.9432 with TTA 11 ( increasing TTA to 23 got 0.9422)\nTrying now on 512x512

", "votes": 0, "replies": [] }, { "id": 946636, "author_name": "SumanSudhir", "author_url": "", "post_date": "2020-07-26T17:34:17.070000", "content": "

EfficientNet B2 384x384\nSingle Fold\nCV 92.9, LB 93.5 using TTA

", "votes": 0, "replies": [] }, { "id": 945213, "author_name": "Yann Majewski", "author_url": "", "post_date": "2020-07-25T16:52:00.303000", "content": "

Something interesting i've experienced: my CV score between image sizes stays the same, around 0.9375 single fold.\nHowever my lb is different depending on the img size: 300x300 0.9266, 380x380 0.9310, 512x512 0.9379. These were all trained with the same setup (Effnetb4) and fold.

", "votes": 0, "replies": [ { "id": 945444, "author_name": "Signal", "author_url": "", "post_date": "2020-07-25T20:06:50.637000", "content": "

@yannmajewski did you train it with one size and then feed it the other sizes, or did you train it each time with each of these sizes?

", "votes": 0, "replies": [] }, { "id": 945598, "author_name": "Yann Majewski", "author_url": "", "post_date": "2020-07-26T02:05:09.597000", "content": "

@brianfeeny each time its a different experiment with a new model!

", "votes": 0, "replies": [] } ] }, { "id": 944908, "author_name": "Ian Pan", "author_url": "", "post_date": "2020-07-25T12:51:07.360000", "content": "

Using the same seed, I can usually get pretty consistent results with the same run. For this dataset, though, I'm getting as much as +/-0.005 on the same fold with the same setup. (This is PyTorch, btw)

", "votes": 0, "replies": [ { "id": 945166, "author_name": "Optimo", "author_url": "", "post_date": "2020-07-25T15:57:09.603000", "content": "

pytorch allows you to have reproducible results, even with GPU computation, so if you fix every random seed you should have exactly the same results, you probably have some free seed somewhere that creates this.

", "votes": 0, "replies": [] } ] }, { "id": 942614, "author_name": "datasaurus", "author_url": "", "post_date": "2020-07-23T21:45:14.667000", "content": "

It's amazing seeing the huge models and image sizes in this thread. I think I need to fit a nitrous bottle to my computer to keep up with you guys 🏎

", "votes": 0, "replies": [] }, { "id": 936080, "author_name": "Rex Parsons", "author_url": "", "post_date": "2020-07-20T00:42:36.767000", "content": "

Thanks for the useful info and discussion that it's started. Could you please share information on which augmentations you used in PyTorch and how you selected them? Thanks!

", "votes": 0, "replies": [] }, { "id": 936072, "author_name": "Jacek Poplawski", "author_url": "", "post_date": "2020-07-20T00:21:58.263000", "content": "

finally got some stability\n256x256, efficientnet-b0, single fold\ntrained on local GPU\nLB 0.9176 \nlocal 0.9166

", "votes": 0, "replies": [] }, { "id": 935927, "author_name": "", "author_url": "", "post_date": "2020-07-19T19:30:10.830000", "content": "

LB : 0.944 Effb4\nImg size: 512x512\nWith external data, no TTA :)

", "votes": 0, "replies": [ { "id": 936093, "author_name": "Dracarys", "author_url": "", "post_date": "2020-07-20T01:09:08.050000", "content": "

@rohan1602 have u used only image features or meta-features as well?

", "votes": 0, "replies": [] }, { "id": 936199, "author_name": "", "author_url": "", "post_date": "2020-07-20T04:17:51.323000", "content": "", "votes": 0, "replies": [] }, { "id": 936306, "author_name": "", "author_url": "", "post_date": "2020-07-20T06:23:25.637000", "content": "

@rohitsingh9990 The model has both image and meta features as input.\n@synked CV: 93.9..I 'm still working on getting a stable CV.

", "votes": 1, "replies": [] } ] }, { "id": 935888, "author_name": "MhdSharuk", "author_url": "", "post_date": "2020-07-19T18:44:20.413000", "content": "

192x192 with no meta data on triple stratified tfrecords, LB score 9.449, cv 0.937

", "votes": 0, "replies": [ { "id": 936124, "author_name": "Jacek Poplawski", "author_url": "", "post_date": "2020-07-20T02:28:14.703000", "content": "

what network arch?

", "votes": 0, "replies": [] }, { "id": 936197, "author_name": "MhdSharuk", "author_url": "", "post_date": "2020-07-20T04:10:02.117000", "content": "

Seresnet50

", "votes": 0, "replies": [] }, { "id": 943369, "author_name": "Vishnu Subramanian", "author_url": "", "post_date": "2020-07-24T10:14:56.607000", "content": "

That sounds really good.

", "votes": 0, "replies": [] } ] }, { "id": 935839, "author_name": "", "author_url": "", "post_date": "2020-07-19T17:37:16.713000", "content": "", "votes": 0, "replies": [] }, { "id": 972704, "author_name": "", "author_url": "", "post_date": "2020-08-16T18:40:33.013000", "content": "", "votes": 0, "replies": [] }, { "id": 972177, "author_name": "", "author_url": "", "post_date": "2020-08-16T09:55:34.827000", "content": "", "votes": -2, "replies": [] }, { "id": 964601, "author_name": "", "author_url": "", "post_date": "2020-08-10T03:20:30.977000", "content": "", "votes": 0, "replies": [] }, { "id": 935837, "author_name": "", "author_url": "", "post_date": "2020-07-19T17:36:22.870000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "935509": "I know one can achieve +0.955 on LB by tweaking a bit some great public TF kernels like [this one](https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords) or [this one](https://www.kaggle.com/agentauers/incredible-tpus-finetune-effnetb0-b6-at-once). But for now I want to experiment as much as possible with Pytorch. \n\nAfter many exepriments with Pytorch, I could achieve 0.944 with 5 folds efficientnetB5 noisy student with some heavy augmentations and trained on @cdeotte 384*384 jpeg files (with external data)\n\nI wonder if someone at the top managed to achieve +0.95/+0.96 with a Pytorch model ? \n\nAre TPU + TFRecords ( allowing huge models + big resolutions and big batch_size trained in short amount of time) the only way to achieve +0.96 ? :)\n\n\nEdit : the model achieve 0.9496 on LB with 15 TTA for each fold. \n", "939957": "I got >0.95 with a single model which is of similar size as your model and after adding some sauce to it (model wise, no LB probing) I got 0.96 from a single model. But you should not take every LB result as is, the test set is small and very un-balanced with very low positive examples. a high Public LB score can turn into a low Private LB. If I judge by my CV std - **Shakeup is Coming** ", "946965": "Note: Everyone is reporting their CV here. We should also report how our CV is setup because without knowing what data people are adding to their validation folds and whether people are using triple stratified or not, we cannot compare CVs.\n\nFor example, if you use external data, don't stratify, don't remove duplicate images, and allow external data in your validation fold, it is easy to get 5 KFold CV AUC 0.98+ \n\n[1]: https://www.kaggle.com/nroman/melanoma-pytorch-starter-efficientnet?scriptVersionId=35185525", "974360": "Could not tune effnet b5, my best is b4 with LB 0.9474, and a bit over 95 with a blend.\n\nI did not blend with any public kernels, couldn't resolve to it. There is a limit to what I am ready to do to get Kaggle points, LOL.\n\nI regret I forgot to try two ideas I had but5 overall it was a great learning experience. Maybe 10 years form now I'll be competitive in computer vision ;) Maybe I should just buy code, I see some gave pointers to relevant online stores...", "972452": "effnetb4, images only, tta, CV 9440 LB 9452\n\nI started with b5 but CV and LB are similar, a bit lower. Trying to tune for a last run ...\n\nI think my final rank will be similar to my current rank.\n\n", "935732": "0.9458 effnet-b2, 512x512, 5 fold with 20xTTA, ext data + metadata, local CV 0.925\n\n@serigne have you tried TTA? if you have some heavy augmentation it could work.", "937048": "0.9424 on LB and 0.9407 on Validation\nSingle Eff B5 Only\nImage Dimension 456x456. Used external data\nPytorch", "936694": "Thanks @optimo \n\nI have run 15 TTA for one fold ..\n\nit improved the score from 0.9293 to 0.9387. \n\nI will update the overall score once it's done for all 5 folds. ", "974507": "@serigne congrats on the good finish.", "936159": "Actually I'm curious, why do you think we see such a difference between pytorch and tf? why pytorch seems to be giving lower scores? Anyone can give some hints?", "935915": "FYI: https://www.kaggle.com/c/liverpool-ion-switching/discussion/145256\nIn the past liverpool competition, TF performs better than PyTorch. Psi found it might be the initialization issue in PyTorch. ", "935754": "Thank you guys !\n\nYou give me motivation to do experiments with 512x512. \n@optimo not that much TTA , just one horizontal flipping. \n\nI have save my best checkpoints fortunately. I will give it a try. ", "935558": "Thanks for voicing out what I wanted to ask as well. Seems like more and more competitions are geared towards TF + TPU \nBy the way, is your 0.944 EfficientnetB5 your LB or your local CV? I'm using @cdeotte csv fold too but the validation score barely hit 0.9. I think it was discussed in his notebook too", "944090": "Guys what's the best CV (Not LB) you got using image only data . For starter mine is 0.9304\n[UPDATE] Mine is 0.938 now.", "939292": "With the same config and 512x512, I got 0.9475 for just the first fold : validation AUC : +0.93.\n\nThings look promising for the 5 folds running :) ", "936991": "EfficientNet b4 on 384x384 jpgs by Chris.\nNo external data,no tta.\nLB - 9216 ", "936191": "PyTorch\nLB : 0.921\nModel : Efficientnet-b0\nImage size : 256x256\nAugmentations : Uniform Augment([paper](https://arxiv.org/abs/2003.14348))\nNo Meta data,\nNo External data,\nTTA\n--> [notebook](https://www.kaggle.com/ttt2209181/melanoma-classification-with-uniform-augment-x256/notebook)\n\nwhen I sized up images to 512 with b3, I got 0.935LB.", "935705": "0.944, 5-fold EfficientNet-B4, 512x512, trained on single 1080Ti ", "935638": "My single model score is only 0.9478.\nNo more feature engineering is built.", "938290": "EfficientNetB6, Image size 384x384, 5-fold stratified with external data\nUsing metafeatures: CV: 0.9217 LB: 0.9409\nWithout metafeatures: CV: 0.9202 LB: 0.9355\nUsing PyTorch", "972666": "EfficientNet-B4, 256x256, 5-Fold, LB 0.9316\n\nBtw what do you mean by \"[...] 15 TTA for each fold.\"\n15 Augmentations blended together with the original image? That seems a lot to me.", "956304": "effb5 on 384x384 got 0.9432 with TTA 11 ( increasing TTA to 23 got 0.9422)\nTrying now on 512x512 ", "946636": "EfficientNet B2 384x384\nSingle Fold\nCV 92.9, LB 93.5 using TTA", "945213": "Something interesting i've experienced: my CV score between image sizes stays the same, around 0.9375 single fold.\nHowever my lb is different depending on the img size: 300x300 0.9266, 380x380 0.9310, 512x512 0.9379. These were all trained with the same setup (Effnetb4) and fold.", "944908": "Using the same seed, I can usually get pretty consistent results with the same run. For this dataset, though, I'm getting as much as +/-0.005 on the same fold with the same setup. (This is PyTorch, btw)", "942614": "It's amazing seeing the huge models and image sizes in this thread. I think I need to fit a nitrous bottle to my computer to keep up with you guys 🏎 ", "936080": "Thanks for the useful info and discussion that it's started. Could you please share information on which augmentations you used in PyTorch and how you selected them? Thanks!", "936072": "finally got some stability\n256x256, efficientnet-b0, single fold\ntrained on local GPU\nLB 0.9176 \nlocal 0.9166", "935927": "LB : 0.944 Effb4\nImg size: 512x512\nWith external data, no TTA :)", "935888": "192x192 with no meta data on triple stratified tfrecords, LB score 9.449, cv 0.937", "935839": "20X TTA!! Good lord. I have rarely seen such high TTA rounds. Thanks for the info though, could be an interesting experiment for me.", "972704": "", "972177": "", "964601": "", "935837": "" } }