After conducting several experiments, I have observed that overfitting may sometimes result in a better score in the leaderboard of this competition. However, it's essential to note that this could be due to the training data's similarity with the test dataset, which may not necessarily be an accurate representation of the unseen data.
\nAn overfitted model with a high accuracy score in training may memorize the noise, thus reducing the loss and performing well on the test set if the noise is consistent between both datasets. However, it may not generalize well to new, unseen data.
\nThis poses an important question, would you submit an overfitted model with a good LB score or a more conservative model that may not perform as well but has a better chance of generalizing to new data?
\nThis decision is critical as it impacts how you configure various hyperparameters if you use transformers, such as the learning rate, weight decay, dropout, and number of epochs. It's important to strike a balance between a good LB score and the model's ability to generalize to new, unseen data.
", "messages": [ { "id": "2231079", "postDate": "04/23/2023 04:43:51", "content": "After conducting several experiments, I have observed that overfitting may sometimes result in a better score in the leaderboard of this competition. However, it's essential to note that this could be due to the training data's similarity with the test dataset, which may not necessarily be an accurate representation of the unseen data.
\nAn overfitted model with a high accuracy score in training may memorize the noise, thus reducing the loss and performing well on the test set if the noise is consistent between both datasets. However, it may not generalize well to new, unseen data.
\nThis poses an important question, would you submit an overfitted model with a good LB score or a more conservative model that may not perform as well but has a better chance of generalizing to new data?
\nThis decision is critical as it impacts how you configure various hyperparameters if you use transformers, such as the learning rate, weight decay, dropout, and number of epochs. It's important to strike a balance between a good LB score and the model's ability to generalize to new, unseen data.
", "rawMarkdown": "After conducting several experiments, I have observed that overfitting may sometimes result in a better score in the leaderboard of this competition. However, it's essential to note that this could be due to the training data's similarity with the test dataset, which may not necessarily be an accurate representation of the unseen data.\n\nAn overfitted model with a high accuracy score in training may memorize the noise, thus reducing the loss and performing well on the test set if the noise is consistent between both datasets. However, it may not generalize well to new, unseen data.\n\nThis poses an important question, would you submit an overfitted model with a good LB score or a more conservative model that may not perform as well but has a better chance of generalizing to new data?\n\nThis decision is critical as it impacts how you configure various hyperparameters if you use transformers, such as the learning rate, weight decay, dropout, and number of epochs. It's important to strike a balance between a good LB score and the model's ability to generalize to new, unseen data.", "votes": null }, { "id": "2231211", "postDate": "04/23/2023 06:37:28", "content": "One thing to consider -
\n\n\n\"There are new participant IDs in the test set. However, your submission notebook does not have access to the participant IDs for any given video.\"
\n
see https://www.kaggle.com/competitions/asl-signs/discussion/390510#2171517
\nDepending on CV approach, information may leak for a participant id in training and to the public LB.
\nIn some testing it seemed the public LB was biased to right hand as well.
But you have 2 submission to select. Some advise to use best of LB and best of your own model choice.
", "rawMarkdown": "One thing to consider -\n\n>\"There are new participant IDs in the test set. However, your submission notebook does not have access to the participant IDs for any given video.\"\n\nsee https://www.kaggle.com/competitions/asl-signs/discussion/390510#2171517\n\nDepending on CV approach, information may leak for a participant id in training and to the public LB. \nIn some testing it seemed the public LB was biased to right hand as well.\n\nBut you have 2 submission to select. Some advise to use best of LB and best of your own model choice.", "votes": null }, { "id": "2231611", "postDate": "04/23/2023 13:22:59", "content": "This is a good question and you can check out this article on how to resolve the feature space inconsistency between the source and target domain.
\nhttps://arxiv.org/pdf/1802.03601.pdf
One thing to consider -
\n\n\n\"There are new participant IDs in the test set. However, your submission notebook does not have access to the participant IDs for any given video.\"
\n
see https://www.kaggle.com/competitions/asl-signs/discussion/390510#2171517
\nDepending on CV approach, information may leak for a participant id in training and to the public LB.
\nIn some testing it seemed the public LB was biased to right hand as well.
But you have 2 submission to select. Some advise to use best of LB and best of your own model choice.
", "votes": null, "replies": [] }, { "id": 2231611, "author_name": "mewmlelswm", "author_url": "", "post_date": "04/23/2023 13:22:59", "content": "This is a good question and you can check out this article on how to resolve the feature space inconsistency between the source and target domain.
\nhttps://arxiv.org/pdf/1802.03601.pdf