In my Transformer model, I have noticed that the loss on the dev set (part of train data that the model hasn't seen) is about 1.15 (see this link https://www.kaggle.com/code/viktorcikojevic/transfomer-model-with-pretrained-model?scriptVersionId=121196850), but the submission score I get is equal to about 1.48. This raises the question of whether there is a discrepancy between the train and test data. I am curious if anyone else has experienced a similar discrepancy and if there are any potential explanations for it? Thanks!
", "messages": [ { "id": "2171095", "postDate": "03/06/2023 14:27:27", "content": "In my Transformer model, I have noticed that the loss on the dev set (part of train data that the model hasn't seen) is about 1.15 (see this link https://www.kaggle.com/code/viktorcikojevic/transfomer-model-with-pretrained-model?scriptVersionId=121196850), but the submission score I get is equal to about 1.48. This raises the question of whether there is a discrepancy between the train and test data. I am curious if anyone else has experienced a similar discrepancy and if there are any potential explanations for it? Thanks!
", "rawMarkdown": "In my Transformer model, I have noticed that the loss on the dev set (part of train data that the model hasn't seen) is about 1.15 (see this link https://www.kaggle.com/code/viktorcikojevic/transfomer-model-with-pretrained-model?scriptVersionId=121196850), but the submission score I get is equal to about 1.48. This raises the question of whether there is a discrepancy between the train and test data. I am curious if anyone else has experienced a similar discrepancy and if there are any potential explanations for it? Thanks!", "votes": null }, { "id": "2171164", "postDate": "03/06/2023 14:56:24", "content": "From my personal experience, train and test performances are really close. Did you properly split your dataset during training ? Have you tested your model on a (not train) batch on your side ?
\nAnother explanation could be your postprocessing ?
\nBest
Check out this post. Perhaps it will help.
", "rawMarkdown": "Check out this post. Perhaps it will help.", "votes": null }, { "id": "2173209", "postDate": "03/08/2023 07:35:18", "content": "Great, thank you, I implemented that after seeing your post! The problem is gone now
", "rawMarkdown": "Great, thank you, I implemented that after seeing your post! The problem is gone now", "votes": null }, { "id": "2173210", "postDate": "03/08/2023 07:38:12", "content": "It's fixed now, the issue seemed to be the processing of the submission, thanks! I implemented sorting by event_id (check this post), and I also allowed for multiple batch_id values in the test set.
From my personal experience, train and test performances are really close. Did you properly split your dataset during training ? Have you tested your model on a (not train) batch on your side ?
\nAnother explanation could be your postprocessing ?
\nBest
It's fixed now, the issue seemed to be the processing of the submission, thanks! I implemented sorting by event_id (check this post), and I also allowed for multiple batch_id values in the test set.
Check out this post. Perhaps it will help.
", "votes": null, "replies": [ { "id": 2173209, "author_name": "viktorcikojevic", "author_url": "", "post_date": "03/08/2023 07:35:18", "content": "Great, thank you, I implemented that after seeing your post! The problem is gone now
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2171095": "In my Transformer model, I have noticed that the loss on the dev set (part of train data that the model hasn't seen) is about 1.15 (see this link https://www.kaggle.com/code/viktorcikojevic/transfomer-model-with-pretrained-model?scriptVersionId=121196850), but the submission score I get is equal to about 1.48. This raises the question of whether there is a discrepancy between the train and test data. I am curious if anyone else has experienced a similar discrepancy and if there are any potential explanations for it? Thanks!", "2171164": "From my personal experience, train and test performances are really close. Did you properly split your dataset during training ? Have you tested your model on a (not train) batch on your side ? \nAnother explanation could be your postprocessing ?\nBest", "2171301": "Check out this post. Perhaps it will help.", "2173209": "Great, thank you, I implemented that after seeing your post! The problem is gone now", "2173210": "It's fixed now, the issue seemed to be the processing of the submission, thanks! I implemented sorting by `event_id` (check [this post](https://www.kaggle.com/competitions/icecube-neutrinos-in-deep-ice/discussion/384967)), and I also allowed for multiple `batch_id` values in the test set." }, "source": "meta" }