![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F113660%2F5f52b7cc831f1b85b152bc9db84ce4ca%2FSelection_028.png?generation=1590838302850389&alt=media\")
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F113660%2F0a3c88207f717e3a4da66ddeb211a830%2FSelection_027.png?generation=1590838304097927&alt=media\")
Here are my initial experiments and corresponding CV vs LB correlation. Hope someone finds it interesting.
\n\ninput 512x512 4fold efficientnets - no TTA \nUPDATES\n2020-06-01 - no metadata, no external images \n| arch | CV | LB |\n| --- | --- | --- |\n| B0 | 0.894 | 0.918 |\n| B3 | 0.899 | 0.921 |\n| B5 | 0.916 | 0.914 |\n| mean | - | 0.926 |\n| + metadata | - | 0.930 |
", "messages": [ { "id": "866652", "postDate": "05/29/2020 15:16:09", "content": "Here are my initial experiments and corresponding CV vs LB correlation. Hope someone finds it interesting.
\n\ninput 512x512 4fold efficientnets - no TTA \nUPDATES\n2020-06-01 - no metadata, no external images \n| arch | CV | LB |\n| --- | --- | --- |\n| B0 | 0.894 | 0.918 |\n| B3 | 0.899 | 0.921 |\n| B5 | 0.916 | 0.914 |\n| mean | - | 0.926 |\n| + metadata | - | 0.930 |
", "rawMarkdown": "Here are my initial experiments and corresponding CV vs LB correlation. Hope someone finds it interesting. \n\ninput 512x512 4fold efficientnets - no TTA \nUPDATES\n2020-06-01 - no metadata, no external images \n| arch | CV | LB |\n| --- | --- | --- |\n| B0 | 0.894 | 0.918 |\n| B3 | 0.899 | 0.921 |\n| B5 | 0.916 | 0.914 |\n| mean | - | 0.926 |\n| + metadata | - | 0.930 |", "votes": null }, { "id": "866703", "postDate": "05/29/2020 16:05:40", "content": "Hi @valanm,\nThanks for creating the thread!
\n\nA few questions:\n- when you say 4 fold CV = 0.894, is this the mean AUC over the 4 folds or is this the out of fold AUC from the 4 folds?\n- when you give your LB score, is this from averaged predictions of each of your 4 models (1 for each fold)? This is important to measure the impact of averaging.
\n\nTo participate to the thread, with a very simple model with inputs 128*128 and a single random fold 80/20:\nFold AUC : 0.877\nLB : 0.872
", "rawMarkdown": "Hi @valanm,\nThanks for creating the thread!\n\nA few questions:\n- when you say 4 fold CV = 0.894, is this the mean AUC over the 4 folds or is this the out of fold AUC from the 4 folds?\n- when you give your LB score, is this from averaged predictions of each of your 4 models (1 for each fold)? This is important to measure the impact of averaging.\n\nTo participate to the thread, with a very simple model with inputs 128*128 and a single random fold 80/20:\nFold AUC : 0.877\nLB : 0.872", "votes": null }, { "id": "866792", "postDate": "05/29/2020 17:10:27", "content": "512 input 4 folds effB5\nCV 0.916\nLB 0.914
", "rawMarkdown": "512 input 4 folds effB5\nCV 0.916\nLB 0.914", "votes": null }, { "id": "867494", "postDate": "05/30/2020 11:07:18", "content": "Thanks! I guess this explains the uplift in LB.
", "rawMarkdown": "Thanks! I guess this explains the uplift in LB.", "votes": null }, { "id": "867511", "postDate": "05/30/2020 11:31:38", "content": "@valanm how many epochs did you train each fold for?\nThanks in Advnace:)
", "rawMarkdown": "valanm how many epochs did you train each fold for?\nThanks in Advnace:)", "votes": null }, { "id": "867520", "postDate": "05/30/2020 11:38:25", "content": "as you know, there is much class imbalance (in the ratio 1 +ve sample to 50 -ve sample). in fact, i wouldn't call it \"imbalance class\". instead i would call it \"out of sample\", it is more like anomaly detection.
\n\nthe ROC and distribution curve would look like this:
\n\n\n
```\nthe figures are for:\nsingle fold efficientnet-b3 for validation set:
\n\nvalid_dataset :
\n len = 2000
\n mode = valid
\n split = split/random_01/fold2_valid_2000.pickle
\n target =
\n 0 1957 (0.979)
\n 1 43 (0.021)
augment = ['null', 'flip']
\nlog loss = 0.095239
\nauc = 0.924980
\ntpr, fpr = 0.744186, 0.074093 at threshold = 0.1
\neer = 0.125703 @ 0.034497
public LB = 0.910\n```
\n\nthe low +ve sample count makes the AUC metric quite unstable. the distribution curve of the +ve samples is not smooth. hence:
\n\ncare needed to be taken to interpret relationship between LB and local cv AUC. it is true that higher local AUC will \"always\" lead to better public LB? and will private and public LB differs?
besides local AUC, is there other metric that is more correlated the public/private LB?
my results of 3 quick models I made:
\n\nCV1 0.8981\nCV2 0.8910\nCV3 0.8963
\n\nLB1 0.889\nLB2 0.886\nLB3 0.907
\n\nsimple blend:\nCV: 0.9164 LB 0.889
\n\nIgnore LB in this competition (assuming train/test follows same distribution)
", "rawMarkdown": "my results of 3 quick models I made:\n\nCV1 0.8981\nCV2 0.8910\nCV3 0.8963\n\nLB1 0.889\nLB2 0.886\nLB3 0.907\n\nsimple blend:\nCV: 0.9164 LB 0.889\n\nIgnore LB in this competition (assuming train/test follows same distribution)", "votes": null }, { "id": "869808", "postDate": "06/01/2020 09:52:58", "content": "single model, 3fold, cv:0925, lb: 0.939\nupdate.\n- single model, 3fold, cv:0930, lb: 0.942
", "rawMarkdown": "single model, 3fold, cv:0925, lb: 0.939\nupdate.\n- single model, 3fold, cv:0930, lb: 0.942", "votes": null }, { "id": "869828", "postDate": "06/01/2020 10:03:48", "content": "images only or with all available data?
", "rawMarkdown": "images only or with all available data?", "votes": null }, { "id": "869833", "postDate": "06/01/2020 10:08:31", "content": "image and metadata
", "rawMarkdown": "image and metadata", "votes": null }, { "id": "869848", "postDate": "06/01/2020 10:14:35", "content": "i haven't done meta myself yet so wonder how much it helps (my experiments are on images only - updated post)
", "rawMarkdown": "i haven't done meta myself yet so wonder how much it helps (my experiments are on images only - updated post)", "votes": null }, { "id": "869943", "postDate": "06/01/2020 11:46:21", "content": "May I ask what GPU you are using and what batch size your able to train with for 512x512 and effB5?
\n\n512x512 effB0 takes 10Gb of GPU Mem with BS=16 on my machine. It seems a bit too much hence the question.
", "rawMarkdown": "May I ask what GPU you are using and what batch size your able to train with for 512x512 and effB5?\n\n512x512 effB0 takes 10Gb of GPU Mem with BS=16 on my machine. It seems a bit too much hence the question.", "votes": null }, { "id": "869948", "postDate": "06/01/2020 11:51:04", "content": "I used free TPU through kaggle kernels
", "rawMarkdown": "I used free TPU through kaggle kernels", "votes": null }, { "id": "869951", "postDate": "06/01/2020 11:51:58", "content": "amazing
", "rawMarkdown": "amazing", "votes": null }, { "id": "869979", "postDate": "06/01/2020 12:15:14", "content": "Ok thanks and what batch size can this handle?
", "rawMarkdown": "Ok thanks and what batch size can this handle?", "votes": null }, { "id": "871110", "postDate": "06/02/2020 06:37:48", "content": "EffNet B3, trained only on 582 images each class\nCV 0.73 LB 0.701
", "rawMarkdown": "EffNet B3, trained only on 582 images each class\nCV 0.73 LB 0.701", "votes": null }, { "id": "871364", "postDate": "06/02/2020 09:44:43", "content": "UPDATE: adding meta as in this notebook helps.
", "rawMarkdown": "UPDATE: adding meta as in this [notebook](https://www.kaggle.com/cdeotte/image-and-tabular-data-0-915) helps.", "votes": null }, { "id": "871714", "postDate": "06/02/2020 15:33:56", "content": "Great job Val. If people want to recreate this in Kaggle notebooks using GPU or TPU, I uploaded TFRecords containing 512x512 here. Then afterward, you can add augmentation and/or meta data. Rotation augmentation for TFRecords is explained here. And CutMix and MixUp here. And the meta data is contained inside my 512x512 TFRecords explained here
", "rawMarkdown": "Great job Val. If people want to recreate this in Kaggle notebooks using GPU or TPU, I uploaded TFRecords containing 512x512 [here][1]. Then afterward, you can add augmentation and/or meta data. Rotation augmentation for TFRecords is explained [here][2]. And CutMix and MixUp [here][3]. And the meta data is contained inside my 512x512 TFRecords explained [here][4]\n\n[1]: https://www.kaggle.com/cdeotte/melanoma-512x512\n[2]: https://www.kaggle.com/c/flower-classification-with-tpus/discussion/132191\n[3]: https://www.kaggle.com/c/flower-classification-with-tpus/discussion/132935\n[4]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/155579", "votes": null }, { "id": "874251", "postDate": "06/04/2020 18:44:44", "content": "Efficientnet B3 - 5fold\nCV: 0.936\nLB: 0.933
", "rawMarkdown": "Efficientnet B3 - 5fold\nCV: 0.936\nLB: 0.933", "votes": null }, { "id": "879496", "postDate": "06/09/2020 14:14:27", "content": "When I experimented with 512x512 effB3 on TPU, it could handle batch size of up to 512 (64*8).
", "rawMarkdown": "When I experimented with 512x512 effB3 on TPU, it could handle batch size of up to 512 (64*8).", "votes": null }, { "id": "879710", "postDate": "06/09/2020 16:50:24", "content": "wow this is huge!
", "rawMarkdown": "wow this is huge!", "votes": null } ], "comments": [ { "id": 866703, "author_name": "optimo", "author_url": "", "post_date": "05/29/2020 16:05:40", "content": "Hi @valanm,\nThanks for creating the thread!
\n\nA few questions:\n- when you say 4 fold CV = 0.894, is this the mean AUC over the 4 folds or is this the out of fold AUC from the 4 folds?\n- when you give your LB score, is this from averaged predictions of each of your 4 models (1 for each fold)? This is important to measure the impact of averaging.
\n\nTo participate to the thread, with a very simple model with inputs 128*128 and a single random fold 80/20:\nFold AUC : 0.877\nLB : 0.872
", "votes": null, "replies": [ { "id": 866792, "author_name": "valanm", "author_url": "", "post_date": "05/29/2020 17:10:27", "content": "Thanks! I guess this explains the uplift in LB.
", "votes": null, "replies": [] }, { "id": 867511, "author_name": "", "author_url": "", "post_date": "05/30/2020 11:31:38", "content": "@valanm how many epochs did you train each fold for?\nThanks in Advnace:)
", "votes": null, "replies": [] } ] }, { "id": 867341, "author_name": "valanm", "author_url": "", "post_date": "05/30/2020 07:20:06", "content": "512 input 4 folds effB5\nCV 0.916\nLB 0.914
", "votes": null, "replies": [ { "id": 869943, "author_name": "optimo", "author_url": "", "post_date": "06/01/2020 11:46:21", "content": "May I ask what GPU you are using and what batch size your able to train with for 512x512 and effB5?
\n\n512x512 effB0 takes 10Gb of GPU Mem with BS=16 on my machine. It seems a bit too much hence the question.
", "votes": null, "replies": [] }, { "id": 869948, "author_name": "valanm", "author_url": "", "post_date": "06/01/2020 11:51:04", "content": "I used free TPU through kaggle kernels
", "votes": null, "replies": [] }, { "id": 869979, "author_name": "optimo", "author_url": "", "post_date": "06/01/2020 12:15:14", "content": "Ok thanks and what batch size can this handle?
", "votes": null, "replies": [] }, { "id": 879496, "author_name": "andrejrybr", "author_url": "", "post_date": "06/09/2020 14:14:27", "content": "When I experimented with 512x512 effB3 on TPU, it could handle batch size of up to 512 (64*8).
", "votes": null, "replies": [] }, { "id": 879710, "author_name": "optimo", "author_url": "", "post_date": "06/09/2020 16:50:24", "content": "wow this is huge!
", "votes": null, "replies": [] } ] }, { "id": 867520, "author_name": "hengck23", "author_url": "", "post_date": "05/30/2020 11:38:25", "content": "as you know, there is much class imbalance (in the ratio 1 +ve sample to 50 -ve sample). in fact, i wouldn't call it \"imbalance class\". instead i would call it \"out of sample\", it is more like anomaly detection.
\n\nthe ROC and distribution curve would look like this:
\n\n\n
```\nthe figures are for:\nsingle fold efficientnet-b3 for validation set:
\n\nvalid_dataset :
\n len = 2000
\n mode = valid
\n split = split/random_01/fold2_valid_2000.pickle
\n target =
\n 0 1957 (0.979)
\n 1 43 (0.021)
augment = ['null', 'flip']
\nlog loss = 0.095239
\nauc = 0.924980
\ntpr, fpr = 0.744186, 0.074093 at threshold = 0.1
\neer = 0.125703 @ 0.034497
public LB = 0.910\n```
\n\nthe low +ve sample count makes the AUC metric quite unstable. the distribution curve of the +ve samples is not smooth. hence:
\n\ncare needed to be taken to interpret relationship between LB and local cv AUC. it is true that higher local AUC will \"always\" lead to better public LB? and will private and public LB differs?
besides local AUC, is there other metric that is more correlated the public/private LB?
my results of 3 quick models I made:
\n\nCV1 0.8981\nCV2 0.8910\nCV3 0.8963
\n\nLB1 0.889\nLB2 0.886\nLB3 0.907
\n\nsimple blend:\nCV: 0.9164 LB 0.889
\n\nIgnore LB in this competition (assuming train/test follows same distribution)
", "votes": null, "replies": [] }, { "id": 869808, "author_name": "phalanx", "author_url": "", "post_date": "06/01/2020 09:52:58", "content": "single model, 3fold, cv:0925, lb: 0.939\nupdate.\n- single model, 3fold, cv:0930, lb: 0.942
", "votes": null, "replies": [ { "id": 869828, "author_name": "valanm", "author_url": "", "post_date": "06/01/2020 10:03:48", "content": "images only or with all available data?
", "votes": null, "replies": [] }, { "id": 869833, "author_name": "phalanx", "author_url": "", "post_date": "06/01/2020 10:08:31", "content": "image and metadata
", "votes": null, "replies": [] }, { "id": 869848, "author_name": "valanm", "author_url": "", "post_date": "06/01/2020 10:14:35", "content": "i haven't done meta myself yet so wonder how much it helps (my experiments are on images only - updated post)
", "votes": null, "replies": [] } ] }, { "id": 869951, "author_name": "neel8800", "author_url": "", "post_date": "06/01/2020 11:51:58", "content": "amazing
", "votes": null, "replies": [] }, { "id": 871110, "author_name": "p4rallax", "author_url": "", "post_date": "06/02/2020 06:37:48", "content": "EffNet B3, trained only on 582 images each class\nCV 0.73 LB 0.701
", "votes": null, "replies": [] }, { "id": 871364, "author_name": "valanm", "author_url": "", "post_date": "06/02/2020 09:44:43", "content": "UPDATE: adding meta as in this notebook helps.
", "votes": null, "replies": [] }, { "id": 871714, "author_name": "cdeotte", "author_url": "", "post_date": "06/02/2020 15:33:56", "content": "Great job Val. If people want to recreate this in Kaggle notebooks using GPU or TPU, I uploaded TFRecords containing 512x512 here. Then afterward, you can add augmentation and/or meta data. Rotation augmentation for TFRecords is explained here. And CutMix and MixUp here. And the meta data is contained inside my 512x512 TFRecords explained here
", "votes": null, "replies": [] }, { "id": 874251, "author_name": "", "author_url": "", "post_date": "06/04/2020 18:44:44", "content": "Efficientnet B3 - 5fold\nCV: 0.936\nLB: 0.933
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "866652": "Here are my initial experiments and corresponding CV vs LB correlation. Hope someone finds it interesting. \n\ninput 512x512 4fold efficientnets - no TTA \nUPDATES\n2020-06-01 - no metadata, no external images \n| arch | CV | LB |\n| --- | --- | --- |\n| B0 | 0.894 | 0.918 |\n| B3 | 0.899 | 0.921 |\n| B5 | 0.916 | 0.914 |\n| mean | - | 0.926 |\n| + metadata | - | 0.930 |", "866703": "Hi @valanm,\nThanks for creating the thread!\n\nA few questions:\n- when you say 4 fold CV = 0.894, is this the mean AUC over the 4 folds or is this the out of fold AUC from the 4 folds?\n- when you give your LB score, is this from averaged predictions of each of your 4 models (1 for each fold)? This is important to measure the impact of averaging.\n\nTo participate to the thread, with a very simple model with inputs 128*128 and a single random fold 80/20:\nFold AUC : 0.877\nLB : 0.872", "866792": "1. CV = mean AUC over the 4 folds\n2. Yes.", "867341": "512 input 4 folds effB5\nCV 0.916\nLB 0.914", "867494": "Thanks! I guess this explains the uplift in LB.", "867511": "valanm how many epochs did you train each fold for?\nThanks in Advnace:)", "867520": "as you know, there is much class imbalance (in the ratio 1 +ve sample to 50 -ve sample). in fact, i wouldn't call it \"imbalance class\". instead i would call it \"out of sample\", it is more like anomaly detection.\n\nthe ROC and distribution curve would look like this:\n\n\n\n\n\n```\nthe figures are for:\nsingle fold efficientnet-b3 for validation set:\n\nvalid_dataset : \t\t\t\t\t\n\tlen = 2000\t\t\t\t\n\tmode = valid\t\t\t\t\n\tsplit = split/random_01/fold2_valid_2000.pickle\t\t\t\t\n\ttarget =\t\t\t\t\n\t\t0 1957 (0.979)\t\t\t\n\t\t1 43 (0.021)\t\t\t\n\n#------------------------------------------------\n\naugment = ['null', 'flip']\t\t\t\nlog loss = 0.095239\t\t\t\nauc = 0.924980\t\t\t\ntpr, fpr = 0.744186, 0.074093 at threshold = 0.1\t\t\t\neer = 0.125703 @ 0.034497\t\t\t\n\npublic LB = 0.910\n```\n\nthe low +ve sample count makes the AUC metric quite unstable. the distribution curve of the +ve samples is not smooth. hence:\n\n1. care needed to be taken to interpret relationship between LB and local cv AUC. it is true that higher local AUC will \"always\" lead to better public LB? and will private and public LB differs?\n\n2. besides local AUC, is there other metric that is more correlated the public/private LB?", "867787": "my results of 3 quick models I made:\n\nCV1 0.8981\nCV2 0.8910\nCV3 0.8963\n\nLB1 0.889\nLB2 0.886\nLB3 0.907\n\nsimple blend:\nCV: 0.9164 LB 0.889\n\nIgnore LB in this competition (assuming train/test follows same distribution)", "869808": "single model, 3fold, cv:0925, lb: 0.939\nupdate.\n- single model, 3fold, cv:0930, lb: 0.942", "869828": "images only or with all available data?", "869833": "image and metadata", "869848": "i haven't done meta myself yet so wonder how much it helps (my experiments are on images only - updated post)", "869943": "May I ask what GPU you are using and what batch size your able to train with for 512x512 and effB5?\n\n512x512 effB0 takes 10Gb of GPU Mem with BS=16 on my machine. It seems a bit too much hence the question.", "869948": "I used free TPU through kaggle kernels", "869951": "amazing", "869979": "Ok thanks and what batch size can this handle?", "871110": "EffNet B3, trained only on 582 images each class\nCV 0.73 LB 0.701", "871364": "UPDATE: adding meta as in this [notebook](https://www.kaggle.com/cdeotte/image-and-tabular-data-0-915) helps.", "871714": "Great job Val. If people want to recreate this in Kaggle notebooks using GPU or TPU, I uploaded TFRecords containing 512x512 [here][1]. Then afterward, you can add augmentation and/or meta data. Rotation augmentation for TFRecords is explained [here][2]. And CutMix and MixUp [here][3]. And the meta data is contained inside my 512x512 TFRecords explained [here][4]\n\n[1]: https://www.kaggle.com/cdeotte/melanoma-512x512\n[2]: https://www.kaggle.com/c/flower-classification-with-tpus/discussion/132191\n[3]: https://www.kaggle.com/c/flower-classification-with-tpus/discussion/132935\n[4]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/155579", "874251": "Efficientnet B3 - 5fold\nCV: 0.936\nLB: 0.933", "879496": "When I experimented with 512x512 effB3 on TPU, it could handle batch size of up to 512 (64*8).", "879710": "wow this is huge!" }, "source": "meta" }