![\"Here\"](\"https://i.ibb.co/xSFkB6W/image-2021-09-20-T19-11-22-574-Z.png\")
is a figure that shows the Loss, accuracy and AUC curves of my model. You can see that the maximum validation and training AUC is arrived to ~85% but when submitting, my score is 57%! Hello every one. After challenging for a long time with the dataset and trying different models, I have finally found a model that is being trained well (based on Training & Validation accuracy and AUC curves). But when I submit the result to the competition, the score is not comparable. I am confused that why is it happening?
\n is a figure that shows the Loss, accuracy and AUC curves of my model. You can see that the maximum validation and training AUC is arrived to ~85% but when submitting, my score is 57%!
Any opinion would be appreciated!
", "messages": [ { "id": "1519807", "postDate": "09/21/2021 23:51:39", "content": "Hello every one. After challenging for a long time with the dataset and trying different models, I have finally found a model that is being trained well (based on Training & Validation accuracy and AUC curves). But when I submit the result to the competition, the score is not comparable. I am confused that why is it happening?
\n is a figure that shows the Loss, accuracy and AUC curves of my model. You can see that the maximum validation and training AUC is arrived to ~85% but when submitting, my score is 57%!
Any opinion would be appreciated!
", "rawMarkdown": "Hello every one. After challenging for a long time with the dataset and trying different models, I have finally found a model that is being trained well (based on Training & Validation accuracy and AUC curves). But when I submit the result to the competition, the score is not comparable. I am confused that why is it happening? \n\n is a figure that shows the Loss, accuracy and AUC curves of my model. You can see that the maximum validation and training AUC is arrived to ~85% but when submitting, my score is 57%! \n\nAny opinion would be appreciated!", "votes": null }, { "id": "1519889", "postDate": "09/22/2021 02:48:32", "content": "@niloofarrahmani , Some notes below.
\nhi ,
\nThe error is being caused because the public lb is only being measured on 87 samples not so good to be honest , so when you run inference on 87 samples which is very low , the metrics are also not that equivalent to what you measured over more samples , it can be more it can be less . But dont worry trust your cv or the score you are seeing , the private lb has 400 samples , in the private lb we will see a huge shake . I have a question to ask why is your validation loss same always ?
Hmm. Good point. But still if you have a good model that generalizes well; why would it only be bad on these 87 cases ?
\nPerhaps it is possible and I am missing something ?
The testing set may have some differences from valid . I would recommend you to split the data into a custom test set , sometimes people overfit to valid set . Thus three sets train valid and test
", "rawMarkdown": "The testing set may have some differences from valid . I would recommend you to split the data into a custom test set , sometimes people overfit to valid set . Thus three sets train valid and test", "votes": null }, { "id": "1520037", "postDate": "09/22/2021 05:31:22", "content": "also if you are using something like 2d cnn then you should measure CV not only parallely for each slice also how you are predicting a single prediction for all slices
", "rawMarkdown": "also if you are using something like 2d cnn then you should measure CV not only parallely for each slice also how you are predicting a single prediction for all slices", "votes": null }, { "id": "1520504", "postDate": "09/22/2021 11:08:58", "content": "Your validation loss is quite high, and it rises during the training. This can be a cause of low score on LB. In fact, I'm not sure if validation AUC is a good metric here.
", "rawMarkdown": "Your validation loss is quite high, and it rises during the training. This can be a cause of low score on LB. In fact, I'm not sure if validation AUC is a good metric here.", "votes": null }, { "id": "1520841", "postDate": "09/22/2021 16:04:17", "content": "Thanks for your reply.
\nActually I haven't use CV fold yet and I should consider it. Also I am using valid_AUC now, maybe valid_loss will be more helpful.
Yes Araik you are right. I also can't get why is that happening. Some time before I had a same problem because of different augmentation on validation and train dataset, but now I am sure this not occurring here. why do you think is it happening?
", "rawMarkdown": "Yes Araik you are right. I also can't get why is that happening. Some time before I had a same problem because of different augmentation on validation and train dataset, but now I am sure this not occurring here. why do you think is it happening?", "votes": null }, { "id": "1521094", "postDate": "09/22/2021 21:55:27", "content": "This is one of the challenges and mysteries of this competition. We have very few rows of data to train on - fewer than 600 records. And when we have few rows of data, weird stuff happens.
\nFor example, you uniquely need to make sure you are not overfitting to TWO datasets: your training dataset AND your validation set. Every tweak you make to your model and retrain, you learn a little bit more information on how to fit better to the validation set…do it too much, and you can risk overfitting. If you try changing the seed, do you still get the amazing AUC?
\nAnother thing that most have noticed in this competition is that the AUC may improve, but the Log Loss gets worse. Log Loss is a smooth metric - it's measuring how good are the predictions you make , like how far away they are from the targets 0 and 1. AUC is not smooth - it is just measuring the ranking of your predictions. So your predictions could all suck, but if they're ranked correctly, you'll get a great AUC.
\nAdd to these problems that the public LB is fewer than 100 examples and we have a really tough competition on our hands. Since the public LB is small, we don't know if a bad public LB is because of the small dataset size, or if it's because our model doesn't generalize well. This is a problem that everyone is battling with right now.
\nMost people recommend to trust your cross-validation over the public LB… but when you have such few amount of training rows, is even the CV trustworthy? We will find out on the private LB.
", "rawMarkdown": "This is one of the challenges and mysteries of this competition. We have very few rows of data to train on - fewer than 600 records. And when we have few rows of data, weird stuff happens. \n\nFor example, you uniquely need to make sure you are not overfitting to TWO datasets: your training dataset AND your validation set. Every tweak you make to your model and retrain, you learn a little bit more information on how to fit better to the validation set...do it too much, and you can risk overfitting. If you try changing the seed, do you still get the amazing AUC?\n\nAnother thing that most have noticed in this competition is that the AUC may improve, but the Log Loss gets worse. Log Loss is a smooth metric - it's measuring how good are the predictions you make , like how far away they are from the targets 0 and 1. AUC is not smooth - it is just measuring the ranking of your predictions. So your predictions could all suck, but if they're ranked correctly, you'll get a great AUC.\n\nAdd to these problems that the public LB is fewer than 100 examples and we have a really tough competition on our hands. Since the public LB is small, we don't know if a bad public LB is because of the small dataset size, or if it's because our model doesn't generalize well. This is a problem that everyone is battling with right now.\n\nMost people recommend to trust your cross-validation over the public LB... but when you have such few amount of training rows, is even the CV trustworthy? We will find out on the private LB.", "votes": null }, { "id": "1521362", "postDate": "09/23/2021 06:52:43", "content": "Your comment clearly shows the problem we are facing in this competition.
\nIt is possible that my model is already over-fitting to a particular Fold and MRI type. The first Fold's FLAIR CV improves, but the others tend to stay the same or get worse. And when we change the seed, the scenery changes completely.
\nAnd I'm interested in what we can learn from this competition in the end. I'm new to working with voxel data and that alone is valuable to me, but if there is a way to do robust and consistent experiments with less data, I would like to learn from the solutions after the competition is over.
I think the model simply doesn't learn anything. It's the same case for me.
", "rawMarkdown": "I think the model simply doesn't learn anything. It's the same case for me.", "votes": null } ], "comments": [ { "id": 1519889, "author_name": "mpsampat", "author_url": "", "post_date": "09/22/2021 02:48:32", "content": "@niloofarrahmani , Some notes below.
\nThanks for your reply.
\nActually I haven't use CV fold yet and I should consider it. Also I am using valid_AUC now, maybe valid_loss will be more helpful.
hi ,
\nThe error is being caused because the public lb is only being measured on 87 samples not so good to be honest , so when you run inference on 87 samples which is very low , the metrics are also not that equivalent to what you measured over more samples , it can be more it can be less . But dont worry trust your cv or the score you are seeing , the private lb has 400 samples , in the private lb we will see a huge shake . I have a question to ask why is your validation loss same always ?
Hmm. Good point. But still if you have a good model that generalizes well; why would it only be bad on these 87 cases ?
\nPerhaps it is possible and I am missing something ?
The testing set may have some differences from valid . I would recommend you to split the data into a custom test set , sometimes people overfit to valid set . Thus three sets train valid and test
", "votes": null, "replies": [] }, { "id": 1520037, "author_name": "swaralipibose", "author_url": "", "post_date": "09/22/2021 05:31:22", "content": "also if you are using something like 2d cnn then you should measure CV not only parallely for each slice also how you are predicting a single prediction for all slices
", "votes": null, "replies": [] } ] }, { "id": 1520504, "author_name": "atamazian", "author_url": "", "post_date": "09/22/2021 11:08:58", "content": "Your validation loss is quite high, and it rises during the training. This can be a cause of low score on LB. In fact, I'm not sure if validation AUC is a good metric here.
", "votes": null, "replies": [ { "id": 1521029, "author_name": "niloofarrahmani", "author_url": "", "post_date": "09/22/2021 20:02:31", "content": "Yes Araik you are right. I also can't get why is that happening. Some time before I had a same problem because of different augmentation on validation and train dataset, but now I am sure this not occurring here. why do you think is it happening?
", "votes": null, "replies": [] }, { "id": 1523022, "author_name": "atamazian", "author_url": "", "post_date": "09/24/2021 21:07:12", "content": "I think the model simply doesn't learn anything. It's the same case for me.
", "votes": null, "replies": [] } ] }, { "id": 1521094, "author_name": "returnofsputnik", "author_url": "", "post_date": "09/22/2021 21:55:27", "content": "This is one of the challenges and mysteries of this competition. We have very few rows of data to train on - fewer than 600 records. And when we have few rows of data, weird stuff happens.
\nFor example, you uniquely need to make sure you are not overfitting to TWO datasets: your training dataset AND your validation set. Every tweak you make to your model and retrain, you learn a little bit more information on how to fit better to the validation set…do it too much, and you can risk overfitting. If you try changing the seed, do you still get the amazing AUC?
\nAnother thing that most have noticed in this competition is that the AUC may improve, but the Log Loss gets worse. Log Loss is a smooth metric - it's measuring how good are the predictions you make , like how far away they are from the targets 0 and 1. AUC is not smooth - it is just measuring the ranking of your predictions. So your predictions could all suck, but if they're ranked correctly, you'll get a great AUC.
\nAdd to these problems that the public LB is fewer than 100 examples and we have a really tough competition on our hands. Since the public LB is small, we don't know if a bad public LB is because of the small dataset size, or if it's because our model doesn't generalize well. This is a problem that everyone is battling with right now.
\nMost people recommend to trust your cross-validation over the public LB… but when you have such few amount of training rows, is even the CV trustworthy? We will find out on the private LB.
", "votes": null, "replies": [ { "id": 1521362, "author_name": "atsunorifujita", "author_url": "", "post_date": "09/23/2021 06:52:43", "content": "Your comment clearly shows the problem we are facing in this competition.
\nIt is possible that my model is already over-fitting to a particular Fold and MRI type. The first Fold's FLAIR CV improves, but the others tend to stay the same or get worse. And when we change the seed, the scenery changes completely.
\nAnd I'm interested in what we can learn from this competition in the end. I'm new to working with voxel data and that alone is valuable to me, but if there is a way to do robust and consistent experiments with less data, I would like to learn from the solutions after the competition is over.