{ "id": 172003, "title": "Higher Score.....!!!", "url": "/competitions/siim-isic-melanoma-classification/discussion/172003", "author_name": "MhdSharuk", "post_date": "2020-08-03T10:13:32.598000", "votes": 14, "comment_count": 40, "views": 0, "content": "

Has anyone reached AUC of 0.95 and above with only a single model with or without meta-data??

", "messages": [ { "id": 956201, "postDate": "2020-08-03T10:13:32.600Z", "content": "

Has anyone reached AUC of 0.95 and above with only a single model with or without meta-data??

", "rawMarkdown": "Has anyone reached AUC of 0.95 and above with only a single model with or without meta-data??", "votes": 13 }, { "id": 956727, "postDate": "2020-08-03T18:21:46.940Z", "content": "

My best single model has an LB score of 0.9613. The corresponding CV is 0.93785 (all numbers are given after TTA). The LB/CV gap is huge, so I am not super-excited about this score and almost certain that it is a result of an overfit. I feel that betting on a single model in this competition would be extremely risky. I think we should pay more attention to our ensemble CV/LB score. Even for those, it is not clear if we can beat the variance present in the data: see Chris's arguments in the following discussion topic: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/171525

", "rawMarkdown": "My best single model has an LB score of 0.9613. The corresponding CV is 0.93785 (all numbers are given after TTA). The LB/CV gap is huge, so I am not super-excited about this score and almost certain that it is a result of an overfit. I feel that betting on a single model in this competition would be extremely risky. I think we should pay more attention to our ensemble CV/LB score. Even for those, it is not clear if we can beat the variance present in the data: see Chris's arguments in the following discussion topic: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/171525", "votes": 12, "replies": [ { "id": 956947, "postDate": "2020-08-04T00:46:42.590Z", "content": "

Great job Alexey. That's a strong single model.

", "rawMarkdown": "Great job Alexey. That's a strong single model.", "votes": 1 }, { "id": 956972, "postDate": "2020-08-04T01:36:22.890Z", "content": "

Thank you Chris! Actually I managed to improve my single model LB to 0.9646 (the 5-fold CV is 0.93938). Still very skeptical about its fate on the private LB. Will keep working on my ensemble. Interestingly enough, my best ensemble has a smaller LB but a higher CV.

", "rawMarkdown": "Thank you Chris! Actually I managed to improve my single model LB to 0.9646 (the 5-fold CV is 0.93938). Still very skeptical about its fate on the private LB. Will keep working on my ensemble. Interestingly enough, my best ensemble has a smaller LB but a higher CV. ", "votes": 2 }, { "id": 957101, "postDate": "2020-08-04T04:03:23.497Z", "content": "

Are you using only image data??

", "rawMarkdown": "Are you using only image data??", "votes": 1 }, { "id": 957671, "postDate": "2020-08-04T13:34:33.163Z", "content": "

Thanks Alexey for sharing this info and thanks Chris for your awesome public kernels in this competition.

\n\n

If I may ask, how are you trying to address the variance in LB scores? \nOne of my models - predict test 5 \"different\" times using TTA (LB score varies from .943 to .949) CV has very less variance (0.002). Also random relation between CV and LB (high CV low LB, lower CV high LB)

\n\n

I am perplexed.

", "rawMarkdown": "Thanks Alexey for sharing this info and thanks Chris for your awesome public kernels in this competition.\n\nIf I may ask, how are you trying to address the variance in LB scores? \nOne of my models - predict test 5 \"different\" times using TTA (LB score varies from .943 to .949) CV has very less variance (0.002). Also random relation between CV and LB (high CV low LB, lower CV high LB)\n\nI am perplexed.", "votes": 1 }, { "id": 957787, "postDate": "2020-08-04T15:01:50.827Z", "content": "

@shivam17818 \n> Are you using only image data??

\n\n

No.

\n\n

@manjeshg03 \n> If I may ask, how are you trying to address the variance in LB scores?

\n\n

By praying? 😲 More seriously, I think creating an ensemble of multiple models might help.

", "rawMarkdown": "@shivam17818 \n> Are you using only image data??\n\nNo.\n\n@manjeshg03 \n> If I may ask, how are you trying to address the variance in LB scores?\n\nBy praying? 😲 More seriously, I think creating an ensemble of multiple models might help.", "votes": 2 } ] }, { "id": 957280, "postDate": "2020-08-04T07:40:56.757Z", "content": "

My best single model cv has 0.956 with all @cdeotte extra data, and best fold can get 0.971,but the corresponding LB score is not very high

", "rawMarkdown": "My best single model cv has 0.956 with all @cdeotte extra data, and best fold can get 0.971,but the corresponding LB score is not very high", "votes": 3, "replies": [ { "id": 957795, "postDate": "2020-08-04T15:06:51.240Z", "content": "

@skgone123 This is a great CV score! Congrats! Are you including any of the external data in your validation set?

", "rawMarkdown": "@skgone123 This is a great CV score! Congrats! Are you including any of the external data in your validation set?", "votes": 1 }, { "id": 960306, "postDate": "2020-08-06T09:48:48.487Z", "content": "

@skgone123 Just wondering, if it is comfortable with you, what do you mean by \"all...extra data?\" To the extent of my knowledge Chris published 2019/2018/2017 data and 580 scraped ISIC images. Do you mean all of these data? I thought that including new-2019 data (especially those out-of-distribution ones) deteriorates LB score)

", "rawMarkdown": "@skgone123 Just wondering, if it is comfortable with you, what do you mean by \"all...extra data?\" To the extent of my knowledge Chris published 2019/2018/2017 data and 580 scraped ISIC images. Do you mean all of these data? I thought that including new-2019 data (especially those out-of-distribution ones) deteriorates LB score)", "votes": 1 }, { "id": 960368, "postDate": "2020-08-06T10:45:57.923Z", "content": "

@graf10a @roguekk007 i used Chris published 2018/2017 and up-sampling data to get best fold score.In validation set didn't include any of the external data, but it is too high

", "rawMarkdown": "@graf10a @roguekk007 i used Chris published 2018/2017 and up-sampling data to get best fold score.In validation set didn't include any of the external data, but it is too high\n", "votes": 1 }, { "id": 960390, "postDate": "2020-08-06T11:00:29.360Z", "content": "

Watch out with using all of Chris his external data! At least part of it contains the positive samples of this competition's dataset. So if you include that, your CV will be very inflated!

", "rawMarkdown": "Watch out with using all of Chris his external data! At least part of it contains the positive samples of this competition's dataset. So if you include that, your CV will be very inflated!", "votes": 2 }, { "id": 960507, "postDate": "2020-08-06T13:09:32.613Z", "content": "

may be, but i didnt check all the external data...

", "rawMarkdown": "may be, but i didnt check all the external data..." }, { "id": 960516, "postDate": "2020-08-06T13:24:37.493Z", "content": "

Tfrecords 0-14 overlap with the training set per his discussion topic. Make sure to exclude those

", "rawMarkdown": "Tfrecords 0-14 overlap with the training set per his discussion topic. Make sure to exclude those", "votes": 1 }, { "id": 960594, "postDate": "2020-08-06T14:25:09.970Z", "content": "

very import information! if its true

", "rawMarkdown": "very import information! if its true", "votes": 1 }, { "id": 960604, "postDate": "2020-08-06T14:32:38.317Z", "content": "
\n

very import information! if its true

\n
\n\n

It is true. My malignant TFRecord00 contains the malignant that are in my 2020 TFRecord00. Then my malignant TFRecord01 contain the malignant that are in my 2020 TFRecord01, etc

\n\n

The first 15 malignant TFRecords numbered 0-14 are the malignant images from 2020 comp data. And the TFRecord numbers match up 0-0, 1-1, 2-2, ..., 13-13, 14-14.

\n\n

Since validation is 2020 data, you must be careful with these first 15 malignant records. You can add all of the other malignant 15-59 to every train fold if you like without leak problem. (There are a total of 60 malignant TFRecords)

", "rawMarkdown": "> very import information! if its true\n\nIt is true. My malignant TFRecord00 contains the malignant that are in my 2020 TFRecord00. Then my malignant TFRecord01 contain the malignant that are in my 2020 TFRecord01, etc\n\nThe first 15 malignant TFRecords numbered 0-14 are the malignant images from 2020 comp data. And the TFRecord numbers match up 0-0, 1-1, 2-2, ..., 13-13, 14-14.\n\nSince validation is 2020 data, you must be careful with these first 15 malignant records. You can add all of the other malignant 15-59 to every train fold if you like without leak problem. (There are a total of 60 malignant TFRecords)", "votes": 4 }, { "id": 960617, "postDate": "2020-08-06T14:40:46.780Z", "content": "

@roguekk007

\n\n

> To the extent of my knowledge Chris published 2019/2018/2017 data and 580 scraped ISIC images. Do you mean all of these data? I thought that including new-2019 data (especially those out-of-distribution ones) deteriorates LB score)

\n\n

In my malignant TFRecords, there are 60 malignant TFRecords total. The first 15 numbered 0-14 are 2020 comp malignant. The next 15 numbered 15-29 are the 580 scraped ISIC. The next 30 have even numbered 30,32, ..., 56,58 with 2018 2017 malignant (2019 \"old portion\"). And finally the odd numbered 31, 33, ..., 57, 59 are 2019 \"new portion\" with weird removed.

\n\n

The 2019 data is \"new portion - originally sized 1024x1024\" and \"old portion - 2018 2017 comp\". The new portion have 2858 malignant. After applying RAPIDS cuML t-SNE, I found that 1185 of these \"new portion\" malignant actually look good. Those are in the odd numbered 31, 33, ..., 57, 59

\n\n

(detailed description of malignant data is here)

", "rawMarkdown": "@roguekk007\n\n> To the extent of my knowledge Chris published 2019/2018/2017 data and 580 scraped ISIC images. Do you mean all of these data? I thought that including new-2019 data (especially those out-of-distribution ones) deteriorates LB score)\n\nIn my malignant TFRecords, there are 60 malignant TFRecords total. The first 15 numbered 0-14 are 2020 comp malignant. The next 15 numbered 15-29 are the 580 scraped ISIC. The next 30 have even numbered 30,32, ..., 56,58 with 2018 2017 malignant (2019 \"old portion\"). And finally the odd numbered 31, 33, ..., 57, 59 are 2019 \"new portion\" with weird removed.\n\nThe 2019 data is \"new portion - originally sized 1024x1024\" and \"old portion - 2018 2017 comp\". The new portion have 2858 malignant. After applying RAPIDS cuML t-SNE, I found that 1185 of these \"new portion\" malignant actually look good. Those are in the odd numbered 31, 33, ..., 57, 59\n\n(detailed description of malignant data is [here][1])\n\n[1]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/169139#945058", "votes": 1 }, { "id": 960638, "postDate": "2020-08-06T15:02:03.123Z", "content": "

thank you for your reply! @cdeotte

", "rawMarkdown": "thank you for your reply! @cdeotte ", "votes": 1 }, { "id": 961264, "postDate": "2020-08-07T03:59:01.150Z", "content": "

@cdeotte Thanks for your reply! Your kind sharing both of data and ideas, have been most helpful in this competition :)

", "rawMarkdown": "@cdeotte Thanks for your reply! Your kind sharing both of data and ideas, have been most helpful in this competition :)", "votes": 1 } ] }, { "id": 956950, "postDate": "2020-08-04T00:50:40.600Z", "content": "

Sure. Here is a single model public notebook (version 4) without meta that scores over LB 0.950. And here is a single model public notebook (version 3) with meta that scores at LB 0.950 with 3 Fold (and conversion to 5 Fold scores over LB 0.950).

\n\n

Public notebooks and ensembles of public notebooks set a high bar to beat in this comp!

", "rawMarkdown": "Sure. Here is a single model public [notebook][1] (version 4) **without** meta that scores over LB 0.950. And here is a single model public [notebook][2] (version 3) **with** meta that scores at LB 0.950 with 3 Fold (and conversion to 5 Fold scores over LB 0.950).\n\nPublic notebooks and ensembles of public notebooks set a high bar to beat in this comp!\n\n[1]: https://www.kaggle.com/ajaykumar7778/efficientnet-cv?scriptVersionId=40043341\n[2]: https://www.kaggle.com/rajnishe/rc-fork-siim-isic-melanoma-384x384?scriptVersionId=39612412", "votes": 3, "replies": [ { "id": 957084, "postDate": "2020-08-04T03:34:17.803Z", "content": "

First of all thanks to you. I've learned a lot from your notebooks and your discussion.\nJust to let you know that your upsample and coarse dropout [notebook] (https://www.kaggle.com/cdeotte/tfrecord-experiments-upsample-and-coarse-dropout) on some hyperparameter change gives me LB score of 0.9517 without any metadata and with meta it's giving 0.9530 and after working on my meta model i got it improved to 0.9547

", "rawMarkdown": "First of all thanks to you. I've learned a lot from your notebooks and your discussion.\nJust to let you know that your upsample and coarse dropout [notebook] (https://www.kaggle.com/cdeotte/tfrecord-experiments-upsample-and-coarse-dropout) on some hyperparameter change gives me LB score of 0.9517 without any metadata and with meta it's giving 0.9530 and after working on my meta model i got it improved to 0.9547", "votes": 1 }, { "id": 957087, "postDate": "2020-08-04T03:36:32.677Z", "content": "

Fantastic. Awesome work @prateek0x

", "rawMarkdown": "Fantastic. Awesome work @prateek0x", "votes": 1 }, { "id": 958192, "postDate": "2020-08-04T20:41:06.650Z", "content": "
\n

Public notebooks and ensembles of public notebooks set a high bar to beat in this comp!

\n
\n\n

They set a high overfitting bar IMHO.

", "rawMarkdown": "> Public notebooks and ensembles of public notebooks set a high bar to beat in this comp!\n\nThey set a high overfitting bar IMHO.", "votes": 11 }, { "id": 958198, "postDate": "2020-08-04T20:46:10.460Z", "content": "

yeah. i looked at the notebooks and their oof results were pretty questionable.

", "rawMarkdown": "yeah. i looked at the notebooks and their oof results were pretty questionable.", "votes": 1 }, { "id": 958588, "postDate": "2020-08-05T04:25:00.327Z", "content": "

@cpmpml How you are accessing that these notebooks are overfitting? I know that the CV LB gap can be useful in evaluating overfitting but is there any other way to find that the predictions are overfitting or not.\n@teeyee314 Can you please tell why do you think so? \nHow to be sure that I am not overfitting the LB?

", "rawMarkdown": "@cpmpml How you are accessing that these notebooks are overfitting? I know that the CV LB gap can be useful in evaluating overfitting but is there any other way to find that the predictions are overfitting or not.\n@teeyee314 Can you please tell why do you think so? \nHow to be sure that I am not overfitting the LB?" }, { "id": 958851, "postDate": "2020-08-05T07:14:04.167Z", "content": "

Because they don't explain how they tune their model. It is then very probable that they used LB feedback to tune.

\n\n

This happens in every competition.

\n\n

The only way to check is to replicate the model using your own cross validation. If you still get good results then it is an exception to what I wrote. And given you have a good CV you don't need the public kernel anymore ;)

\n\n

Using public kernel without double checking CV scores is gambling. You can win, sure, but it is risky. And on average you lose.

\n\n

I wrote \"your own CV\" because one can also overfit to CV.

", "rawMarkdown": "Because they don't explain how they tune their model. It is then very probable that they used LB feedback to tune.\n\nThis happens in every competition.\n\nThe only way to check is to replicate the model using your own cross validation. If you still get good results then it is an exception to what I wrote. And given you have a good CV you don't need the public kernel anymore ;)\n\nUsing public kernel without double checking CV scores is gambling. You can win, sure, but it is risky. And on average you lose.\n\nI wrote \"your own CV\" because one can also overfit to CV. ", "votes": 2 }, { "id": 960857, "postDate": "2020-08-06T18:31:10.900Z", "content": "

I have to agree. The CV in for example this https://www.kaggle.com/iwatatakuya/siim-isic-efficientnet-b6-single-model-lb-0-9475 is usually in the 0.91 when running it and uses the exact same data as I do but for some reason scores higher LB with lower CV than my local version ? I'll follow my local CV across 5 folds and see where that gets me rather than LB score.

\n\n

Anyway with 3 possible selection you can go for different strategies.

", "rawMarkdown": "I have to agree. The CV in for example this https://www.kaggle.com/iwatatakuya/siim-isic-efficientnet-b6-single-model-lb-0-9475 is usually in the 0.91 when running it and uses the exact same data as I do but for some reason scores higher LB with lower CV than my local version ? I'll follow my local CV across 5 folds and see where that gets me rather than LB score.\n\nAnyway with 3 possible selection you can go for different strategies." } ] }, { "id": 958932, "postDate": "2020-08-05T08:15:45.213Z", "content": "

Our best single model in LB is 0.9534 with B3.\nOur best CV model is 0.94x.

", "rawMarkdown": "Our best single model in LB is 0.9534 with B3.\nOur best CV model is 0.94x.", "votes": 4 }, { "id": 961850, "postDate": "2020-08-07T14:59:11.390Z", "content": "

I currently reach 0.9411 with a B2 on 224x224. No meta. I'm using local pytorch so training is a bit slow compared to TF TPU. I'll go B6 384 when I have tried all ideas at that lower resolution. Joined the competition seriously very late so I hope I'll have time to do all I want to do :|

", "rawMarkdown": "I currently reach 0.9411 with a B2 on 224x224. No meta. I'm using local pytorch so training is a bit slow compared to TF TPU. I'll go B6 384 when I have tried all ideas at that lower resolution. Joined the competition seriously very late so I hope I'll have time to do all I want to do :|", "votes": 1, "replies": [ { "id": 961988, "postDate": "2020-08-07T17:27:14.920Z", "content": "

same here. tried to push my gtx 1080ti with b6 384, it can only handle 4 samples at a time, and takes 1 hour for 1 epoch..

", "rawMarkdown": "same here. tried to push my gtx 1080ti with b6 384, it can only handle 4 samples at a time, and takes 1 hour for 1 epoch.. " }, { "id": 961997, "postDate": "2020-08-07T17:37:23.547Z", "content": "

IIRC I can do more than that with mixed precision. But it takes 1hour per epoch :X

", "rawMarkdown": "IIRC I can do more than that with mixed precision. But it takes 1hour per epoch :X" } ] }, { "id": 958562, "postDate": "2020-08-05T03:59:30.510Z", "content": "

If you ensemble models trained on different image sizes with the same model architecture ( for example b6) is this still considered as a single model here?

", "rawMarkdown": "If you ensemble models trained on different image sizes with the same model architecture ( for example b6) is this still considered as a single model here? ", "votes": 1 }, { "id": 956284, "postDate": "2020-08-03T11:42:23.707Z", "content": "

Yep, I've 0.9502 with single image-only model.

", "rawMarkdown": "Yep, I've 0.9502 with single image-only model.", "votes": 1 }, { "id": 956246, "postDate": "2020-08-03T10:59:09.187Z", "content": "

Yes, there are replies on the single model score topics that are > 0.95

", "rawMarkdown": "Yes, there are replies on the single model score topics that are > 0.95", "votes": 1, "replies": [ { "id": 956448, "postDate": "2020-08-03T14:06:09.140Z", "content": "

@group16 Could you post which all that are...

", "rawMarkdown": "@group16 Could you post which all that are...", "votes": -1 } ] }, { "id": 961388, "postDate": "2020-08-07T06:23:15.767Z", "content": "

Our best single model in LB is 0.9533 with B6.
\nOur best CV model is 0.931.
\nWhat remains for us are 2 things Progressive training and playing with these parameters.
\n\"batchnormmomentum\",\"batchnormepsilon\",\"dropoutrate\",\"widthcoefficient\",\"depthcoefficient\",\"depthdivisor\",\"mindepth\",\"dropconnect_rate\".

", "rawMarkdown": "Our best single model in LB is 0.9533 with B6.\nOur best CV model is 0.931.\nWhat remains for us are 2 things Progressive training and playing with these parameters.\n\"batch_norm_momentum\",\"batch_norm_epsilon\",\"dropout_rate\",\"width_coefficient\",\"depth_coefficient\",\"depth_divisor\",\"min_depth\",\"drop_connect_rate\".", "votes": 2 }, { "id": 956417, "postDate": "2020-08-03T13:30:40.647Z", "content": "

0.9548 (LB) my best single model

", "rawMarkdown": "0.9548 (LB) my best single model", "replies": [ { "id": 957982, "postDate": "2020-08-04T17:04:57.427Z", "content": "

me too. However when i ensemble using multiple models then my score decreases. Also combining with XGB did't improve my score

", "rawMarkdown": "me too. However when i ensemble using multiple models then my score decreases. Also combining with XGB did't improve my score" }, { "id": 958044, "postDate": "2020-08-04T18:15:25.160Z", "content": "

Might be a signal you are overfitting OR it is just you are improving the private LB part (it is really hard to say in this competition. I expect a HUGE shakeup in the end and it will be more or less lottery - personal opinion - don't take this as something valid). In that situation I'll try to do stuff that generally would make sense rather than trying to maximise CV/LB.

", "rawMarkdown": "Might be a signal you are overfitting OR it is just you are improving the private LB part (it is really hard to say in this competition. I expect a HUGE shakeup in the end and it will be more or less lottery - personal opinion - don't take this as something valid). In that situation I'll try to do stuff that generally would make sense rather than trying to maximise CV/LB." } ] }, { "id": 961384, "postDate": "2020-08-07T06:13:38.540Z", "rawMarkdown": "", "isDeleted": true }, { "id": 961383, "postDate": "2020-08-07T06:13:38.343Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 956727, "author_name": "Alexey Pronin", "author_url": "", "post_date": "2020-08-03T18:21:46.940000", "content": "

My best single model has an LB score of 0.9613. The corresponding CV is 0.93785 (all numbers are given after TTA). The LB/CV gap is huge, so I am not super-excited about this score and almost certain that it is a result of an overfit. I feel that betting on a single model in this competition would be extremely risky. I think we should pay more attention to our ensemble CV/LB score. Even for those, it is not clear if we can beat the variance present in the data: see Chris's arguments in the following discussion topic: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/171525

", "votes": 12, "replies": [ { "id": 956947, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-04T00:46:42.590000", "content": "

Great job Alexey. That's a strong single model.

", "votes": 1, "replies": [] }, { "id": 956972, "author_name": "Alexey Pronin", "author_url": "", "post_date": "2020-08-04T01:36:22.890000", "content": "

Thank you Chris! Actually I managed to improve my single model LB to 0.9646 (the 5-fold CV is 0.93938). Still very skeptical about its fate on the private LB. Will keep working on my ensemble. Interestingly enough, my best ensemble has a smaller LB but a higher CV.

", "votes": 2, "replies": [] }, { "id": 957101, "author_name": "Shivam", "author_url": "", "post_date": "2020-08-04T04:03:23.497000", "content": "

Are you using only image data??

", "votes": 1, "replies": [] }, { "id": 957671, "author_name": "Manjesh Gupta", "author_url": "", "post_date": "2020-08-04T13:34:33.163000", "content": "

Thanks Alexey for sharing this info and thanks Chris for your awesome public kernels in this competition.

\n\n

If I may ask, how are you trying to address the variance in LB scores? \nOne of my models - predict test 5 \"different\" times using TTA (LB score varies from .943 to .949) CV has very less variance (0.002). Also random relation between CV and LB (high CV low LB, lower CV high LB)

\n\n

I am perplexed.

", "votes": 1, "replies": [] }, { "id": 957787, "author_name": "Alexey Pronin", "author_url": "", "post_date": "2020-08-04T15:01:50.827000", "content": "

@shivam17818 \n> Are you using only image data??

\n\n

No.

\n\n

@manjeshg03 \n> If I may ask, how are you trying to address the variance in LB scores?

\n\n

By praying? 😲 More seriously, I think creating an ensemble of multiple models might help.

", "votes": 2, "replies": [] } ] }, { "id": 957280, "author_name": "Shimei", "author_url": "", "post_date": "2020-08-04T07:40:56.757000", "content": "

My best single model cv has 0.956 with all @cdeotte extra data, and best fold can get 0.971,but the corresponding LB score is not very high

", "votes": 3, "replies": [ { "id": 957795, "author_name": "Alexey Pronin", "author_url": "", "post_date": "2020-08-04T15:06:51.240000", "content": "

@skgone123 This is a great CV score! Congrats! Are you including any of the external data in your validation set?

", "votes": 1, "replies": [] }, { "id": 960306, "author_name": "Nicholas Lyu", "author_url": "", "post_date": "2020-08-06T09:48:48.487000", "content": "

@skgone123 Just wondering, if it is comfortable with you, what do you mean by \"all...extra data?\" To the extent of my knowledge Chris published 2019/2018/2017 data and 580 scraped ISIC images. Do you mean all of these data? I thought that including new-2019 data (especially those out-of-distribution ones) deteriorates LB score)

", "votes": 1, "replies": [] }, { "id": 960368, "author_name": "Shimei", "author_url": "", "post_date": "2020-08-06T10:45:57.923000", "content": "

@graf10a @roguekk007 i used Chris published 2018/2017 and up-sampling data to get best fold score.In validation set didn't include any of the external data, but it is too high

", "votes": 1, "replies": [] }, { "id": 960390, "author_name": "Gilles Vandewiele", "author_url": "", "post_date": "2020-08-06T11:00:29.360000", "content": "

Watch out with using all of Chris his external data! At least part of it contains the positive samples of this competition's dataset. So if you include that, your CV will be very inflated!

", "votes": 2, "replies": [] }, { "id": 960507, "author_name": "Shimei", "author_url": "", "post_date": "2020-08-06T13:09:32.613000", "content": "

may be, but i didnt check all the external data...

", "votes": 0, "replies": [] }, { "id": 960516, "author_name": "Gilles Vandewiele", "author_url": "", "post_date": "2020-08-06T13:24:37.493000", "content": "

Tfrecords 0-14 overlap with the training set per his discussion topic. Make sure to exclude those

", "votes": 1, "replies": [] }, { "id": 960594, "author_name": "Shimei", "author_url": "", "post_date": "2020-08-06T14:25:09.970000", "content": "

very import information! if its true

", "votes": 1, "replies": [] }, { "id": 960604, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-06T14:32:38.317000", "content": "
\n

very import information! if its true

\n
\n\n

It is true. My malignant TFRecord00 contains the malignant that are in my 2020 TFRecord00. Then my malignant TFRecord01 contain the malignant that are in my 2020 TFRecord01, etc

\n\n

The first 15 malignant TFRecords numbered 0-14 are the malignant images from 2020 comp data. And the TFRecord numbers match up 0-0, 1-1, 2-2, ..., 13-13, 14-14.

\n\n

Since validation is 2020 data, you must be careful with these first 15 malignant records. You can add all of the other malignant 15-59 to every train fold if you like without leak problem. (There are a total of 60 malignant TFRecords)

", "votes": 4, "replies": [] }, { "id": 960617, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-06T14:40:46.780000", "content": "

@roguekk007

\n\n

> To the extent of my knowledge Chris published 2019/2018/2017 data and 580 scraped ISIC images. Do you mean all of these data? I thought that including new-2019 data (especially those out-of-distribution ones) deteriorates LB score)

\n\n

In my malignant TFRecords, there are 60 malignant TFRecords total. The first 15 numbered 0-14 are 2020 comp malignant. The next 15 numbered 15-29 are the 580 scraped ISIC. The next 30 have even numbered 30,32, ..., 56,58 with 2018 2017 malignant (2019 \"old portion\"). And finally the odd numbered 31, 33, ..., 57, 59 are 2019 \"new portion\" with weird removed.

\n\n

The 2019 data is \"new portion - originally sized 1024x1024\" and \"old portion - 2018 2017 comp\". The new portion have 2858 malignant. After applying RAPIDS cuML t-SNE, I found that 1185 of these \"new portion\" malignant actually look good. Those are in the odd numbered 31, 33, ..., 57, 59

\n\n

(detailed description of malignant data is here)

", "votes": 1, "replies": [] }, { "id": 960638, "author_name": "Shimei", "author_url": "", "post_date": "2020-08-06T15:02:03.123000", "content": "

thank you for your reply! @cdeotte

", "votes": 1, "replies": [] }, { "id": 961264, "author_name": "Nicholas Lyu", "author_url": "", "post_date": "2020-08-07T03:59:01.150000", "content": "

@cdeotte Thanks for your reply! Your kind sharing both of data and ideas, have been most helpful in this competition :)

", "votes": 1, "replies": [] } ] }, { "id": 956950, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-04T00:50:40.600000", "content": "

Sure. Here is a single model public notebook (version 4) without meta that scores over LB 0.950. And here is a single model public notebook (version 3) with meta that scores at LB 0.950 with 3 Fold (and conversion to 5 Fold scores over LB 0.950).

\n\n

Public notebooks and ensembles of public notebooks set a high bar to beat in this comp!

", "votes": 3, "replies": [ { "id": 957084, "author_name": "Prateek Mishra", "author_url": "", "post_date": "2020-08-04T03:34:17.803000", "content": "

First of all thanks to you. I've learned a lot from your notebooks and your discussion.\nJust to let you know that your upsample and coarse dropout [notebook] (https://www.kaggle.com/cdeotte/tfrecord-experiments-upsample-and-coarse-dropout) on some hyperparameter change gives me LB score of 0.9517 without any metadata and with meta it's giving 0.9530 and after working on my meta model i got it improved to 0.9547

", "votes": 1, "replies": [] }, { "id": 957087, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-04T03:36:32.677000", "content": "

Fantastic. Awesome work @prateek0x

", "votes": 1, "replies": [] }, { "id": 958192, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-04T20:41:06.650000", "content": "
\n

Public notebooks and ensembles of public notebooks set a high bar to beat in this comp!

\n
\n\n

They set a high overfitting bar IMHO.

", "votes": 11, "replies": [] }, { "id": 958198, "author_name": "Tim Yee", "author_url": "", "post_date": "2020-08-04T20:46:10.460000", "content": "

yeah. i looked at the notebooks and their oof results were pretty questionable.

", "votes": 1, "replies": [] }, { "id": 958588, "author_name": "Prateek Mishra", "author_url": "", "post_date": "2020-08-05T04:25:00.327000", "content": "

@cpmpml How you are accessing that these notebooks are overfitting? I know that the CV LB gap can be useful in evaluating overfitting but is there any other way to find that the predictions are overfitting or not.\n@teeyee314 Can you please tell why do you think so? \nHow to be sure that I am not overfitting the LB?

", "votes": 0, "replies": [] }, { "id": 958851, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-05T07:14:04.167000", "content": "

Because they don't explain how they tune their model. It is then very probable that they used LB feedback to tune.

\n\n

This happens in every competition.

\n\n

The only way to check is to replicate the model using your own cross validation. If you still get good results then it is an exception to what I wrote. And given you have a good CV you don't need the public kernel anymore ;)

\n\n

Using public kernel without double checking CV scores is gambling. You can win, sure, but it is risky. And on average you lose.

\n\n

I wrote \"your own CV\" because one can also overfit to CV.

", "votes": 2, "replies": [] }, { "id": 960857, "author_name": "Arnaud Roussel", "author_url": "", "post_date": "2020-08-06T18:31:10.900000", "content": "

I have to agree. The CV in for example this https://www.kaggle.com/iwatatakuya/siim-isic-efficientnet-b6-single-model-lb-0-9475 is usually in the 0.91 when running it and uses the exact same data as I do but for some reason scores higher LB with lower CV than my local version ? I'll follow my local CV across 5 folds and see where that gets me rather than LB score.

\n\n

Anyway with 3 possible selection you can go for different strategies.

", "votes": 0, "replies": [] } ] }, { "id": 958932, "author_name": "Hai Nam Nguyen", "author_url": "", "post_date": "2020-08-05T08:15:45.213000", "content": "

Our best single model in LB is 0.9534 with B3.\nOur best CV model is 0.94x.

", "votes": 4, "replies": [] }, { "id": 961850, "author_name": "Arnaud Roussel", "author_url": "", "post_date": "2020-08-07T14:59:11.390000", "content": "

I currently reach 0.9411 with a B2 on 224x224. No meta. I'm using local pytorch so training is a bit slow compared to TF TPU. I'll go B6 384 when I have tried all ideas at that lower resolution. Joined the competition seriously very late so I hope I'll have time to do all I want to do :|

", "votes": 1, "replies": [ { "id": 961988, "author_name": "yimacs", "author_url": "", "post_date": "2020-08-07T17:27:14.920000", "content": "

same here. tried to push my gtx 1080ti with b6 384, it can only handle 4 samples at a time, and takes 1 hour for 1 epoch..

", "votes": 0, "replies": [] }, { "id": 961997, "author_name": "Arnaud Roussel", "author_url": "", "post_date": "2020-08-07T17:37:23.547000", "content": "

IIRC I can do more than that with mixed precision. But it takes 1hour per epoch :X

", "votes": 0, "replies": [] } ] }, { "id": 958562, "author_name": "josechango", "author_url": "", "post_date": "2020-08-05T03:59:30.510000", "content": "

If you ensemble models trained on different image sizes with the same model architecture ( for example b6) is this still considered as a single model here?

", "votes": 1, "replies": [] }, { "id": 956284, "author_name": "Stanislav Blinov", "author_url": "", "post_date": "2020-08-03T11:42:23.707000", "content": "

Yep, I've 0.9502 with single image-only model.

", "votes": 1, "replies": [] }, { "id": 956246, "author_name": "Gilles Vandewiele", "author_url": "", "post_date": "2020-08-03T10:59:09.187000", "content": "

Yes, there are replies on the single model score topics that are > 0.95

", "votes": 1, "replies": [ { "id": 956448, "author_name": "MhdSharuk", "author_url": "", "post_date": "2020-08-03T14:06:09.140000", "content": "

@group16 Could you post which all that are...

", "votes": -1, "replies": [] } ] }, { "id": 961388, "author_name": "Bs004", "author_url": "", "post_date": "2020-08-07T06:23:15.767000", "content": "

Our best single model in LB is 0.9533 with B6.
\nOur best CV model is 0.931.
\nWhat remains for us are 2 things Progressive training and playing with these parameters.
\n\"batchnormmomentum\",\"batchnormepsilon\",\"dropoutrate\",\"widthcoefficient\",\"depthcoefficient\",\"depthdivisor\",\"mindepth\",\"dropconnect_rate\".

", "votes": 2, "replies": [] }, { "id": 956417, "author_name": "Georgi Pamukov", "author_url": "", "post_date": "2020-08-03T13:30:40.647000", "content": "

0.9548 (LB) my best single model

", "votes": 0, "replies": [ { "id": 957982, "author_name": "khyati sharma", "author_url": "", "post_date": "2020-08-04T17:04:57.427000", "content": "

me too. However when i ensemble using multiple models then my score decreases. Also combining with XGB did't improve my score

", "votes": 0, "replies": [] }, { "id": 958044, "author_name": "Georgi Pamukov", "author_url": "", "post_date": "2020-08-04T18:15:25.160000", "content": "

Might be a signal you are overfitting OR it is just you are improving the private LB part (it is really hard to say in this competition. I expect a HUGE shakeup in the end and it will be more or less lottery - personal opinion - don't take this as something valid). In that situation I'll try to do stuff that generally would make sense rather than trying to maximise CV/LB.

", "votes": 0, "replies": [] } ] }, { "id": 961384, "author_name": "", "author_url": "", "post_date": "2020-08-07T06:13:38.540000", "content": "", "votes": 0, "replies": [] }, { "id": 961383, "author_name": "", "author_url": "", "post_date": "2020-08-07T06:13:38.343000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "956201": "Has anyone reached AUC of 0.95 and above with only a single model with or without meta-data??", "956727": "My best single model has an LB score of 0.9613. The corresponding CV is 0.93785 (all numbers are given after TTA). The LB/CV gap is huge, so I am not super-excited about this score and almost certain that it is a result of an overfit. I feel that betting on a single model in this competition would be extremely risky. I think we should pay more attention to our ensemble CV/LB score. Even for those, it is not clear if we can beat the variance present in the data: see Chris's arguments in the following discussion topic: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/171525", "957280": "My best single model cv has 0.956 with all @cdeotte extra data, and best fold can get 0.971,but the corresponding LB score is not very high", "956950": "Sure. Here is a single model public [notebook][1] (version 4) **without** meta that scores over LB 0.950. And here is a single model public [notebook][2] (version 3) **with** meta that scores at LB 0.950 with 3 Fold (and conversion to 5 Fold scores over LB 0.950).\n\nPublic notebooks and ensembles of public notebooks set a high bar to beat in this comp!\n\n[1]: https://www.kaggle.com/ajaykumar7778/efficientnet-cv?scriptVersionId=40043341\n[2]: https://www.kaggle.com/rajnishe/rc-fork-siim-isic-melanoma-384x384?scriptVersionId=39612412", "958932": "Our best single model in LB is 0.9534 with B3.\nOur best CV model is 0.94x.", "961850": "I currently reach 0.9411 with a B2 on 224x224. No meta. I'm using local pytorch so training is a bit slow compared to TF TPU. I'll go B6 384 when I have tried all ideas at that lower resolution. Joined the competition seriously very late so I hope I'll have time to do all I want to do :|", "958562": "If you ensemble models trained on different image sizes with the same model architecture ( for example b6) is this still considered as a single model here? ", "956284": "Yep, I've 0.9502 with single image-only model.", "956246": "Yes, there are replies on the single model score topics that are > 0.95", "961388": "Our best single model in LB is 0.9533 with B6.\nOur best CV model is 0.931.\nWhat remains for us are 2 things Progressive training and playing with these parameters.\n\"batch_norm_momentum\",\"batch_norm_epsilon\",\"dropout_rate\",\"width_coefficient\",\"depth_coefficient\",\"depth_divisor\",\"min_depth\",\"drop_connect_rate\".", "956417": "0.9548 (LB) my best single model", "961384": "", "961383": "" } }