I have been trying out a lot of different approaches for this competition either ideas that come to my mind or the ones that are shared in discussions thanks to everyone sharing.
\nI used 10 fold random cv for validating my ideas but since it was taking too much time I switched back to 5 fold. Unfortunately, I am not able to find a stable cv strategy that aligns with public LB yet.
\nThere might be several and more things playing role in this:
\n1) Noise in the training set, noise in validation set, noise in public test set - are they random? If yes random validation improvements should reflect to LB. But in my case a model scores 0.910 in CV then score 0.895 in LB and oppositely another model can score 0.895 in CV and 0.901 in LB.
\n2) Variance. It is mentioned in papers robust training is needed in the presence of noise and that might be why a lot of TTA rounds actually help, additionally to blending different models. Some improvements can be lower then the effect of the variance. Here we see that a lot top teams are only 0.001 away from each other which is 16 images approximately. I am not able to tell even if my model is statistically significant or it’s by chance.
\nI am wondering if anyone manage to come up with a solid working validation strategy. So far only thing I can think of is doing kfold multiple times to reduce the variance as much a possible.
", "messages": [ { "id": 1115291, "postDate": "2020-12-16T06:58:48.747Z", "content": "I have been trying out a lot of different approaches for this competition either ideas that come to my mind or the ones that are shared in discussions thanks to everyone sharing.
\nI used 10 fold random cv for validating my ideas but since it was taking too much time I switched back to 5 fold. Unfortunately, I am not able to find a stable cv strategy that aligns with public LB yet.
\nThere might be several and more things playing role in this:
\n1) Noise in the training set, noise in validation set, noise in public test set - are they random? If yes random validation improvements should reflect to LB. But in my case a model scores 0.910 in CV then score 0.895 in LB and oppositely another model can score 0.895 in CV and 0.901 in LB.
\n2) Variance. It is mentioned in papers robust training is needed in the presence of noise and that might be why a lot of TTA rounds actually help, additionally to blending different models. Some improvements can be lower then the effect of the variance. Here we see that a lot top teams are only 0.001 away from each other which is 16 images approximately. I am not able to tell even if my model is statistically significant or it’s by chance.
\nI am wondering if anyone manage to come up with a solid working validation strategy. So far only thing I can think of is doing kfold multiple times to reduce the variance as much a possible.
", "rawMarkdown": "I have been trying out a lot of different approaches for this competition either ideas that come to my mind or the ones that are shared in discussions thanks to everyone sharing.\n\nI used 10 fold random cv for validating my ideas but since it was taking too much time I switched back to 5 fold. Unfortunately, I am not able to find a stable cv strategy that aligns with public LB yet.\n\nThere might be several and more things playing role in this: \n\n1) Noise in the training set, noise in validation set, noise in public test set - are they random? If yes random validation improvements should reflect to LB. But in my case a model scores 0.910 in CV then score 0.895 in LB and oppositely another model can score 0.895 in CV and 0.901 in LB.\n\n2) Variance. It is mentioned in papers robust training is needed in the presence of noise and that might be why a lot of TTA rounds actually help, additionally to blending different models. Some improvements can be lower then the effect of the variance. Here we see that a lot top teams are only 0.001 away from each other which is 16 images approximately. I am not able to tell even if my model is statistically significant or it’s by chance. \n\nI am wondering if anyone manage to come up with a solid working validation strategy. So far only thing I can think of is doing kfold multiple times to reduce the variance as much a possible.\n\n\n\n\n\n\n\n\n\n\n\n\n\n", "votes": 13 }, { "id": 1115548, "postDate": "2020-12-16T11:05:09.760Z", "content": "If the noise is completely random means that I hasn't got the same distribution over the wrong labels in training vs public set. So if you select your model based on the validation set metric, that doesn't mean that the performance on the leaderboard set will be correlated with it
\nThe techniques I am using for benchmark a model configuration are:
\nI agree with both of your points. Robust training + bootstrapping and reducing variance via multiple kfold should be what we need to trust instead of LB.
", "rawMarkdown": "I agree with both of your points. Robust training + bootstrapping and reducing variance via multiple kfold should be what we need to trust instead of LB.", "votes": 1 } ] }, { "id": 1116755, "postDate": "2020-12-17T12:52:09.197Z", "content": "1) Is \"random cv\" StratifiedK Split? My CV and LB difference is quite a lot CV: 0.8905 LB: 0.872. Any comments?
\n2) How do you know there exists a noise in data that is causing issues in correlation btw CV and LB? Maybe the test data is from different city or the split is Time based?
1) Random cv is not stratified, just random kfold.
\n2) I am not sure, it’s just an hypothesis but maybe noise distributions can be different in training and test images.
", "rawMarkdown": "1) Random cv is not stratified, just random kfold. \n\n2) I am not sure, it’s just an hypothesis but maybe noise distributions can be different in training and test images. ", "votes": 2 }, { "id": 1118399, "postDate": "2020-12-19T03:21:36.893Z", "content": "My opinion is no way for 1st to get so bad score on just 5 class classification problem unless noisy dataset.
", "rawMarkdown": "My opinion is no way for 1st to get so bad score on just 5 class classification problem unless noisy dataset." } ] }, { "id": 1116197, "postDate": "2020-12-17T00:22:31.160Z", "content": "Given the small number of experiments we can use a t-distribution for estimating the confidence interval of validation accuracy might be a good idea. That can give a sense of possible shake-up:
\nconfidence_level = 0.95\ndegrees_freedom = sample.size - 1 # sample is an array of validation scores from multiple experiments\nsample_mean = np.mean(sample)\nsample_standard_error = scipy.stats.sem(sample)\n\nconfidence_interval = scipy.stats.t.interval(confidence_level, degrees_freedom, sample_mean, sample_standard_error)\nNot really necessary to use any approximation. See below, you can use the beta distribution to just get the exact Clopper-Pearson CI as Beta(0.025, correct, N-correct+1) to Beta(0.975, correct+1, N-correct). Or some other variant like adding 0.5 such as Beta(0.025, correct+0.5, N-correct+0.5) to Beta(0.975, correct+0.5, N-correct+0.5) - although that will not really make much of a difference.
", "rawMarkdown": "Not really necessary to use any approximation. See below, you can use the beta distribution to just get the exact Clopper-Pearson CI as Beta(0.025, correct, N-correct+1) to Beta(0.975, correct+1, N-correct). Or some other variant like adding 0.5 such as Beta(0.025, correct+0.5, N-correct+0.5) to Beta(0.975, correct+0.5, N-correct+0.5) - although that will not really make much of a difference.", "votes": 1 }, { "id": 1116219, "postDate": "2020-12-17T01:04:02.690Z", "content": "Thanks for pointing it out, I am not much familiar with using beta distribution for CI :) Do you mean with this method we don't even need to run multiple experiments to get a list sample of scores, then to sample mean and stderr calcs? All we need is # of correct predictions and sample size from just 1 experiment?
", "rawMarkdown": "Thanks for pointing it out, I am not much familiar with using beta distribution for CI :) Do you mean with this method we don't even need to run multiple experiments to get a list sample of scores, then to sample mean and stderr calcs? All we need is # of correct predictions and sample size from just 1 experiment?", "votes": 1 }, { "id": 1116815, "postDate": "2020-12-17T13:40:31.607Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1116814, "postDate": "2020-12-17T13:40:31.607Z", "content": "Ooops, I misread your comment somewhat. What I meant more is that you don't need to use a normal approximation to get a confidence interval based on independent images either being correctly or incorrectly classified (i.e. for a single experiment). Once you get to multiple experiments, it actually gets a bit messier, because whether the same image gets correctly classified for different seeds is not independent. If I understand what you did, you e.g. had 6 seeds, so got 6 estimates of accuracy (let's say [0.85, 0.86, 0.84, 0.85, 0.86, 0.85]) and then you look at the mean and variance around that. I guess that kind of will capture the correlation in a sense and is pretty easy to implement, but e.g. ignores how uncertain you are about each of the estimates. I assume it would likely be more efficient (=more information with fewer seeds), if one fit a random effects logistic regression (with a random effect on the intercept for each image and perhaps class as an explanatory variable).
", "rawMarkdown": "Ooops, I misread your comment somewhat. What I meant more is that you don't need to use a normal approximation to get a confidence interval based on independent images either being correctly or incorrectly classified (i.e. for a single experiment). Once you get to multiple experiments, it actually gets a bit messier, because whether the same image gets correctly classified for different seeds is not independent. If I understand what you did, you e.g. had 6 seeds, so got 6 estimates of accuracy (let's say [0.85, 0.86, 0.84, 0.85, 0.86, 0.85]) and then you look at the mean and variance around that. I guess that kind of will capture the correlation in a sense and is pretty easy to implement, but e.g. ignores how uncertain you are about each of the estimates. I assume it would likely be more efficient (=more information with fewer seeds), if one fit a random effects logistic regression (with a random effect on the intercept for each image and perhaps class as an explanatory variable).", "votes": 1 } ] }, { "id": 1115826, "postDate": "2020-12-16T15:47:55.267Z", "content": "You get an approximate idea of uncertainty around accuracy by looking at a credible interval (or a confidence interval) for the accuracy. E.g. for 0.895 accuracy, a 80% CrI goes from 0.892 to 0.898 (while a 99.9% CrI has an upper end at 0.902).
\nfrom scipy.stats import beta\nlcrl = np.round( beta.ppf(0.1, 19150, 2247), 5)\nmedian = np.round( beta.ppf(0.5, 19150, 2247), 5)\nucrl = np.round( beta.ppf(0.9, 19150, 2247), 5)\nprint(f'Median {median} with lower 80% CrI limit {lcrl} to upper 80% CrI limit {ucrl}')\nSo, variation that goes from 0.895 to 0.901 is perhaps a bit surprising, but not completely beyond what you might get through things that are just randomness. Of course, there might be some systematic differences - e.g. the organizers might have split older vs. newer photos.
\nRepeated 5-fold (e.g. 2 times or 3 times 5-fold - as mentioned by @vladvdv) could be an option to reduce the variation (but because there would be a correlation between the out-of-fold validation scores, its a bit harder to say how much that reduces your uncertainty around performance). Of course, that has the same downside as 10 fold (i.e. having to train 10 or 15 times), but I would actually expect it to improve CV to LB correlation more than using a single 10-fold CV scheme.
\nStratified (by class) k-fold (if you are not using it) could be another option (or perhaps stratified by image type).
", "rawMarkdown": "You get an approximate idea of uncertainty around accuracy by looking at a credible interval (or a confidence interval) for the accuracy. E.g. for 0.895 accuracy, a 80% CrI goes from 0.892 to 0.898 (while a 99.9% CrI has an upper end at 0.902).\n```\nfrom scipy.stats import beta\nlcrl = np.round( beta.ppf(0.1, 19150, 2247), 5)\nmedian = np.round( beta.ppf(0.5, 19150, 2247), 5)\nucrl = np.round( beta.ppf(0.9, 19150, 2247), 5)\nprint(f'Median {median} with lower 80% CrI limit {lcrl} to upper 80% CrI limit {ucrl}')\n\n```\nSo, variation that goes from 0.895 to 0.901 is perhaps a bit surprising, but not completely beyond what you might get through things that are just randomness. Of course, there might be some systematic differences - e.g. the organizers might have split older vs. newer photos.\n\nRepeated 5-fold (e.g. 2 times or 3 times 5-fold - as mentioned by @vladvdv) could be an option to reduce the variation (but because there would be a correlation between the out-of-fold validation scores, its a bit harder to say how much that reduces your uncertainty around performance). Of course, that has the same downside as 10 fold (i.e. having to train 10 or 15 times), but I would actually expect it to improve CV to LB correlation more than using a single 10-fold CV scheme.\n\nStratified (by class) k-fold (if you are not using it) could be another option (or perhaps [stratified by image type](https://www.kaggle.com/bjoernholzhauer/cassava-leaf-disease-classif-eda-cv-strategy)).", "votes": 1, "replies": [ { "id": 1116222, "postDate": "2020-12-17T01:05:32.160Z", "content": "Nice kernel thanks for linking it!
", "rawMarkdown": "Nice kernel thanks for linking it!" } ] }, { "id": 1118907, "postDate": "2020-12-19T14:13:11.300Z", "content": "Hello!
\nIs single fold 901?
If the noise is completely random means that I hasn't got the same distribution over the wrong labels in training vs public set. So if you select your model based on the validation set metric, that doesn't mean that the performance on the leaderboard set will be correlated with it
\nThe techniques I am using for benchmark a model configuration are:
\nI agree with both of your points. Robust training + bootstrapping and reducing variance via multiple kfold should be what we need to trust instead of LB.
", "votes": 1, "replies": [] } ] }, { "id": 1116755, "author_name": "ammad", "author_url": "", "post_date": "2020-12-17T12:52:09.197000", "content": "1) Is \"random cv\" StratifiedK Split? My CV and LB difference is quite a lot CV: 0.8905 LB: 0.872. Any comments?
\n2) How do you know there exists a noise in data that is causing issues in correlation btw CV and LB? Maybe the test data is from different city or the split is Time based?
1) Random cv is not stratified, just random kfold.
\n2) I am not sure, it’s just an hypothesis but maybe noise distributions can be different in training and test images.
", "votes": 2, "replies": [] }, { "id": 1118399, "author_name": "Yi Wu", "author_url": "", "post_date": "2020-12-19T03:21:36.893000", "content": "My opinion is no way for 1st to get so bad score on just 5 class classification problem unless noisy dataset.
", "votes": 0, "replies": [] } ] }, { "id": 1116197, "author_name": "Kerem Turgutlu", "author_url": "", "post_date": "2020-12-17T00:22:31.160000", "content": "Given the small number of experiments we can use a t-distribution for estimating the confidence interval of validation accuracy might be a good idea. That can give a sense of possible shake-up:
\nconfidence_level = 0.95\ndegrees_freedom = sample.size - 1 # sample is an array of validation scores from multiple experiments\nsample_mean = np.mean(sample)\nsample_standard_error = scipy.stats.sem(sample)\n\nconfidence_interval = scipy.stats.t.interval(confidence_level, degrees_freedom, sample_mean, sample_standard_error)\nNot really necessary to use any approximation. See below, you can use the beta distribution to just get the exact Clopper-Pearson CI as Beta(0.025, correct, N-correct+1) to Beta(0.975, correct+1, N-correct). Or some other variant like adding 0.5 such as Beta(0.025, correct+0.5, N-correct+0.5) to Beta(0.975, correct+0.5, N-correct+0.5) - although that will not really make much of a difference.
", "votes": 1, "replies": [] }, { "id": 1116219, "author_name": "Kerem Turgutlu", "author_url": "", "post_date": "2020-12-17T01:04:02.690000", "content": "Thanks for pointing it out, I am not much familiar with using beta distribution for CI :) Do you mean with this method we don't even need to run multiple experiments to get a list sample of scores, then to sample mean and stderr calcs? All we need is # of correct predictions and sample size from just 1 experiment?
", "votes": 1, "replies": [] }, { "id": 1116815, "author_name": "", "author_url": "", "post_date": "2020-12-17T13:40:31.607000", "content": "", "votes": 0, "replies": [] }, { "id": 1116814, "author_name": "Björn", "author_url": "", "post_date": "2020-12-17T13:40:31.607000", "content": "Ooops, I misread your comment somewhat. What I meant more is that you don't need to use a normal approximation to get a confidence interval based on independent images either being correctly or incorrectly classified (i.e. for a single experiment). Once you get to multiple experiments, it actually gets a bit messier, because whether the same image gets correctly classified for different seeds is not independent. If I understand what you did, you e.g. had 6 seeds, so got 6 estimates of accuracy (let's say [0.85, 0.86, 0.84, 0.85, 0.86, 0.85]) and then you look at the mean and variance around that. I guess that kind of will capture the correlation in a sense and is pretty easy to implement, but e.g. ignores how uncertain you are about each of the estimates. I assume it would likely be more efficient (=more information with fewer seeds), if one fit a random effects logistic regression (with a random effect on the intercept for each image and perhaps class as an explanatory variable).
", "votes": 1, "replies": [] } ] }, { "id": 1115826, "author_name": "Björn", "author_url": "", "post_date": "2020-12-16T15:47:55.267000", "content": "You get an approximate idea of uncertainty around accuracy by looking at a credible interval (or a confidence interval) for the accuracy. E.g. for 0.895 accuracy, a 80% CrI goes from 0.892 to 0.898 (while a 99.9% CrI has an upper end at 0.902).
\nfrom scipy.stats import beta\nlcrl = np.round( beta.ppf(0.1, 19150, 2247), 5)\nmedian = np.round( beta.ppf(0.5, 19150, 2247), 5)\nucrl = np.round( beta.ppf(0.9, 19150, 2247), 5)\nprint(f'Median {median} with lower 80% CrI limit {lcrl} to upper 80% CrI limit {ucrl}')\nSo, variation that goes from 0.895 to 0.901 is perhaps a bit surprising, but not completely beyond what you might get through things that are just randomness. Of course, there might be some systematic differences - e.g. the organizers might have split older vs. newer photos.
\nRepeated 5-fold (e.g. 2 times or 3 times 5-fold - as mentioned by @vladvdv) could be an option to reduce the variation (but because there would be a correlation between the out-of-fold validation scores, its a bit harder to say how much that reduces your uncertainty around performance). Of course, that has the same downside as 10 fold (i.e. having to train 10 or 15 times), but I would actually expect it to improve CV to LB correlation more than using a single 10-fold CV scheme.
\nStratified (by class) k-fold (if you are not using it) could be another option (or perhaps stratified by image type).
", "votes": 1, "replies": [ { "id": 1116222, "author_name": "Kerem Turgutlu", "author_url": "", "post_date": "2020-12-17T01:05:32.160000", "content": "Nice kernel thanks for linking it!
", "votes": 0, "replies": [] } ] }, { "id": 1118907, "author_name": "HAaHAa", "author_url": "", "post_date": "2020-12-19T14:13:11.300000", "content": "Hello!
\nIs single fold 901?