I'm focused on making a realiable cross-validation (CV). I have experimented with 3, 4 and 5 folds by using a Stratified Gruped K-Fold strategy. Despite I'm getting a similar performance in train and valdations sets, the results obtained for test set are far from leaderboard. What I'm suppose to do? repeat the CV until getting the expected results?
\n\nHere are some of my CV performance locally and their corresponding public ladeboard score:
\n\n| local | public LB |\n|-------|-----------|\n| 0.629 | 0.686 |\n| 0.774 | 0.670 |\n| 0.800 | 0.652 |\n| 0.615 | 0.639 |\n| 0.740 | 0.664 |\n| 0.779 | 0.686 |\n| 0.794 | 0.638 |\n| 0.815 | 0.649 |\n| 0.835 | 0.633 |
\n\nThank you in advance.
", "messages": [ { "id": 918803, "postDate": "2020-07-07T13:59:50.730Z", "content": "The exact same thing is happening to us! We don't know what the problem is, there's apparently no leakage between our training and validation sets (we coded a loop that went though all the arrays on both sets and not two equal images were found). Let me know if you solve the problem, we'll do the same :)
", "rawMarkdown": "The exact same thing is happening to us! We don't know what the problem is, there's apparently no leakage between our training and validation sets (we coded a loop that went though all the arrays on both sets and not two equal images were found). Let me know if you solve the problem, we'll do the same :)", "votes": 2, "replies": [ { "id": 920526, "postDate": "2020-07-08T16:43:49.333Z", "content": "Until the next week I won't be able to try some ideas I have. Probably averagin multiple epochs would produce more estable results. However this approach would affect negatively to the score.
", "rawMarkdown": "Until the next week I won't be able to try some ideas I have. Probably averagin multiple epochs would produce more estable results. However this approach would affect negatively to the score. " }, { "id": 920662, "postDate": "2020-07-08T18:33:06.713Z", "content": "the instability is inherent to the evaluation metric and class imbalance with very few positive examples, a single false negative can have a big effect on the score.
", "rawMarkdown": "the instability is inherent to the evaluation metric and class imbalance with very few positive examples, a single false negative can have a big effect on the score." }, { "id": 921836, "postDate": "2020-07-09T15:41:49.633Z", "content": "i usually use mean of 5 fold. Not just predict of 1 fold. In my opinion, it's better to know about data, not overfit
", "rawMarkdown": "i usually use mean of 5 fold. Not just predict of 1 fold. In my opinion, it's better to know about data, not overfit" } ] }, { "id": 917043, "postDate": "2020-07-06T07:43:55.097Z", "content": "You could have a leak between training and validation and are overfitting the validation set as a result of it.
\n\nI've used standard KFold CV and the results seem pretty stable (from last 5 single model submissions):
\n\n| CV | public LB |\n| --- | --- |\n| 0.928 | 0.924 |\n| 0.915 | 0.915 |\n|0.922 | 0.922 |\n| 0.901 | 0.910 |\n|0.905 | 0.900 |
\n\nOne would expect patient-wise to be more robust.
\n\nEdit: \nIn fact, the model is fairly low performing so maybe it is something else. Could you give a description of the model and if it's a deep learning model could you post a training curve?
", "rawMarkdown": "You could have a leak between training and validation and are overfitting the validation set as a result of it.\n\nI've used standard KFold CV and the results seem pretty stable (from last 5 single model submissions):\n\n| CV | public LB |\n| --- | --- |\n| 0.928 | 0.924 |\n| 0.915 | 0.915 |\n|0.922 | 0.922 |\n| 0.901 | 0.910 |\n|0.905 | 0.900 |\n\nOne would expect patient-wise to be more robust.\n\nEdit: \nIn fact, the model is fairly low performing so maybe it is something else. Could you give a description of the model and if it's a deep learning model could you post a training curve?", "votes": 2, "replies": [ { "id": 917059, "postDate": "2020-07-06T07:55:56.470Z", "content": "I'm training a ResNet-34 on 128x128 images during 3 epochs. I'm also using Adam(1e-3) as optimizer and FocalLoss(2.0, 0.25) as loss function. Moreover, I'm not applying any regularization technique.
\n\nThe CV has been built using sex and *anatom_site_general_challenge* as feature columns as well as \"target\" and \"patient_id\" (following @meemr5 notebook)
", "rawMarkdown": "I'm training a ResNet-34 on 128x128 images during 3 epochs. I'm also using Adam(1e-3) as optimizer and FocalLoss(2.0, 0.25) as loss function. Moreover, I'm not applying any regularization technique.\n\nThe CV has been built using *sex* and *anatom_site_general_challenge* as feature columns as well as \"target\" and \"patient_id\" (following @meemr5 [notebook]( https://www.kaggle.com/meemr5/melanoma-classification-eda-sgkfold-logisticr/notebook#5.-Validation-Strategy---Stratified-Group-K-Fold))" }, { "id": 917087, "postDate": "2020-07-06T08:25:57.253Z", "content": "Thanks, that's useful. Look at your training/validation loss curve and try training for 15 epochs and see if that improves things; I'm guessing the number of epochs is too small.
", "rawMarkdown": "Thanks, that's useful. Look at your training/validation loss curve and try training for 15 epochs and see if that improves things; I'm guessing the number of epochs is too small." }, { "id": 917154, "postDate": "2020-07-06T09:21:31.413Z", "content": "Hey, @cayala!\nI'm using SGKFOLD as you mentioned.
\n\nHere are my CV and LB Score:\n| CV - SGKFOLD |LB |\n| --- | --- |\n|0.8980|0.859|\n| 0.912220621897585| 0.884|\n|0.9165890794989842|0.903|\n|0.9059320092201233|0.910|\n|0.8970596194267273|0.894|
\n\nI'm using TTA-10 for test-set predictions. Can't say where you are making a mistake.
", "rawMarkdown": "Hey, @cayala!\nI'm using SGKFOLD as you mentioned. \n\n---\n\nHere are my CV and LB Score:\n| CV - SGKFOLD |LB |\n| --- | --- |\n|0.8980|0.859|\n| 0.912220621897585| 0.884|\n|0.9165890794989842|0.903|\n|0.9059320092201233|0.910|\n|0.8970596194267273|0.894|\n\nI'm using TTA-10 for test-set predictions. Can't say where you are making a mistake." }, { "id": 917162, "postDate": "2020-07-06T09:26:29.453Z", "rawMarkdown": "", "isDeleted": true }, { "id": 917187, "postDate": "2020-07-06T09:49:19.107Z", "content": "I'm using standard Kfold at the TFRecord level using the records @cdeotte made (here).
\n\nThere are some results I don't understand (e.g., CV : 0.901 and LB 0.910).
\n\nChris may have grouped patients into the same TFRecord file, in that case I would have unintentionally been doing patient-wise cross validation, I will check this today.
\n\nEDIT:
\n\nJust checked, the TFRecords are not separated by patient_id so I am just using standard KFold. Definitely not recommended to use, I see no reason not to use patient-wise CV.
", "rawMarkdown": "I'm using standard Kfold at the TFRecord level using the records @cdeotte made ([here](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/155579)). \n\nThere are some results I don't understand (e.g., CV : 0.901 and LB 0.910).\n\nChris may have grouped patients into the same TFRecord file, in that case I would have unintentionally been doing patient-wise cross validation, I will check this today.\n\nEDIT: \n\nJust checked, the TFRecords are not separated by patient_id so I am just using standard KFold. Definitely not recommended to use, I see no reason not to use patient-wise CV.", "votes": 1 } ] }, { "id": 916984, "postDate": "2020-07-06T06:54:11.343Z", "content": "I'm focused on making a realiable cross-validation (CV). I have experimented with 3, 4 and 5 folds by using a Stratified Gruped K-Fold strategy. Despite I'm getting a similar performance in train and valdations sets, the results obtained for test set are far from leaderboard. What I'm suppose to do? repeat the CV until getting the expected results?
\n\nHere are some of my CV performance locally and their corresponding public ladeboard score:
\n\n| local | public LB |\n|-------|-----------|\n| 0.629 | 0.686 |\n| 0.774 | 0.670 |\n| 0.800 | 0.652 |\n| 0.615 | 0.639 |\n| 0.740 | 0.664 |\n| 0.779 | 0.686 |\n| 0.794 | 0.638 |\n| 0.815 | 0.649 |\n| 0.835 | 0.633 |
\n\nThank you in advance.
", "rawMarkdown": "I'm focused on making a realiable cross-validation (CV). I have experimented with 3, 4 and 5 folds by using a Stratified Gruped K-Fold strategy. Despite I'm getting a similar performance in train and valdations sets, the results obtained for test set are far from leaderboard. What I'm suppose to do? repeat the CV until getting the expected results?\n\nHere are some of my CV performance locally and their corresponding public ladeboard score:\n\n| local | public LB |\n|-------|-----------|\n| 0.629 | 0.686 |\n| 0.774 | 0.670 |\n| 0.800 | 0.652 |\n| 0.615 | 0.639 |\n| 0.740 | 0.664 |\n| 0.779 | 0.686 |\n| 0.794 | 0.638 |\n| 0.815 | 0.649 |\n| 0.835 | 0.633 |\n\nThank you in advance.", "votes": 2 } ], "comments": [ { "id": 918803, "author_name": "Fernando Gaston Codony", "author_url": "", "post_date": "2020-07-07T13:59:50.730000", "content": "The exact same thing is happening to us! We don't know what the problem is, there's apparently no leakage between our training and validation sets (we coded a loop that went though all the arrays on both sets and not two equal images were found). Let me know if you solve the problem, we'll do the same :)
", "votes": 2, "replies": [ { "id": 920526, "author_name": "cayala", "author_url": "", "post_date": "2020-07-08T16:43:49.333000", "content": "Until the next week I won't be able to try some ideas I have. Probably averagin multiple epochs would produce more estable results. However this approach would affect negatively to the score.
", "votes": 0, "replies": [] }, { "id": 920662, "author_name": "alaa", "author_url": "", "post_date": "2020-07-08T18:33:06.713000", "content": "the instability is inherent to the evaluation metric and class imbalance with very few positive examples, a single false negative can have a big effect on the score.
", "votes": 0, "replies": [] }, { "id": 921836, "author_name": "Manh Lab", "author_url": "", "post_date": "2020-07-09T15:41:49.633000", "content": "i usually use mean of 5 fold. Not just predict of 1 fold. In my opinion, it's better to know about data, not overfit
", "votes": 0, "replies": [] } ] }, { "id": 917043, "author_name": "FChmiel", "author_url": "", "post_date": "2020-07-06T07:43:55.097000", "content": "You could have a leak between training and validation and are overfitting the validation set as a result of it.
\n\nI've used standard KFold CV and the results seem pretty stable (from last 5 single model submissions):
\n\n| CV | public LB |\n| --- | --- |\n| 0.928 | 0.924 |\n| 0.915 | 0.915 |\n|0.922 | 0.922 |\n| 0.901 | 0.910 |\n|0.905 | 0.900 |
\n\nOne would expect patient-wise to be more robust.
\n\nEdit: \nIn fact, the model is fairly low performing so maybe it is something else. Could you give a description of the model and if it's a deep learning model could you post a training curve?
", "votes": 2, "replies": [ { "id": 917059, "author_name": "cayala", "author_url": "", "post_date": "2020-07-06T07:55:56.470000", "content": "I'm training a ResNet-34 on 128x128 images during 3 epochs. I'm also using Adam(1e-3) as optimizer and FocalLoss(2.0, 0.25) as loss function. Moreover, I'm not applying any regularization technique.
\n\nThe CV has been built using sex and *anatom_site_general_challenge* as feature columns as well as \"target\" and \"patient_id\" (following @meemr5 notebook)
", "votes": 0, "replies": [] }, { "id": 917087, "author_name": "FChmiel", "author_url": "", "post_date": "2020-07-06T08:25:57.253000", "content": "Thanks, that's useful. Look at your training/validation loss curve and try training for 15 epochs and see if that improves things; I'm guessing the number of epochs is too small.
", "votes": 0, "replies": [] }, { "id": 917154, "author_name": "Meet Ranoliya", "author_url": "", "post_date": "2020-07-06T09:21:31.413000", "content": "Hey, @cayala!\nI'm using SGKFOLD as you mentioned.
\n\nHere are my CV and LB Score:\n| CV - SGKFOLD |LB |\n| --- | --- |\n|0.8980|0.859|\n| 0.912220621897585| 0.884|\n|0.9165890794989842|0.903|\n|0.9059320092201233|0.910|\n|0.8970596194267273|0.894|
\n\nI'm using TTA-10 for test-set predictions. Can't say where you are making a mistake.
", "votes": 0, "replies": [] }, { "id": 917162, "author_name": "", "author_url": "", "post_date": "2020-07-06T09:26:29.453000", "content": "", "votes": 0, "replies": [] }, { "id": 917187, "author_name": "FChmiel", "author_url": "", "post_date": "2020-07-06T09:49:19.107000", "content": "I'm using standard Kfold at the TFRecord level using the records @cdeotte made (here).
\n\nThere are some results I don't understand (e.g., CV : 0.901 and LB 0.910).
\n\nChris may have grouped patients into the same TFRecord file, in that case I would have unintentionally been doing patient-wise cross validation, I will check this today.
\n\nEDIT:
\n\nJust checked, the TFRecords are not separated by patient_id so I am just using standard KFold. Definitely not recommended to use, I see no reason not to use patient-wise CV.
", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "918803": "The exact same thing is happening to us! We don't know what the problem is, there's apparently no leakage between our training and validation sets (we coded a loop that went though all the arrays on both sets and not two equal images were found). Let me know if you solve the problem, we'll do the same :)", "917043": "You could have a leak between training and validation and are overfitting the validation set as a result of it.\n\nI've used standard KFold CV and the results seem pretty stable (from last 5 single model submissions):\n\n| CV | public LB |\n| --- | --- |\n| 0.928 | 0.924 |\n| 0.915 | 0.915 |\n|0.922 | 0.922 |\n| 0.901 | 0.910 |\n|0.905 | 0.900 |\n\nOne would expect patient-wise to be more robust.\n\nEdit: \nIn fact, the model is fairly low performing so maybe it is something else. Could you give a description of the model and if it's a deep learning model could you post a training curve?", "916984": "I'm focused on making a realiable cross-validation (CV). I have experimented with 3, 4 and 5 folds by using a Stratified Gruped K-Fold strategy. Despite I'm getting a similar performance in train and valdations sets, the results obtained for test set are far from leaderboard. What I'm suppose to do? repeat the CV until getting the expected results?\n\nHere are some of my CV performance locally and their corresponding public ladeboard score:\n\n| local | public LB |\n|-------|-----------|\n| 0.629 | 0.686 |\n| 0.774 | 0.670 |\n| 0.800 | 0.652 |\n| 0.615 | 0.639 |\n| 0.740 | 0.664 |\n| 0.779 | 0.686 |\n| 0.794 | 0.638 |\n| 0.815 | 0.649 |\n| 0.835 | 0.633 |\n\nThank you in advance." } }