Hi guys, I'm just trying to get things setup so that my public LB performance tracks my local performance, but having some issues. For example, in some experiments I'm reporting local MAP@5 scores of around 0.50 on my validation set but finding it's much lower on public LB - around 0.25.
\nIs anyone else experiencing the same result? If not, I suspect my local validation checks have a bug in there somewhere. Thanks and good luck to everyone.
", "messages": [ { "id": "1686707", "postDate": "02/12/2022 09:50:20", "content": "Hi guys, I'm just trying to get things setup so that my public LB performance tracks my local performance, but having some issues. For example, in some experiments I'm reporting local MAP@5 scores of around 0.50 on my validation set but finding it's much lower on public LB - around 0.25.
\nIs anyone else experiencing the same result? If not, I suspect my local validation checks have a bug in there somewhere. Thanks and good luck to everyone.
", "rawMarkdown": "Hi guys, I'm just trying to get things setup so that my public LB performance tracks my local performance, but having some issues. For example, in some experiments I'm reporting local MAP@5 scores of around 0.50 on my validation set but finding it's much lower on public LB - around 0.25.\n\nIs anyone else experiencing the same result? If not, I suspect my local validation checks have a bug in there somewhere. Thanks and good luck to everyone.", "votes": null }, { "id": "1686848", "postDate": "02/12/2022 11:40:09", "content": "Do you take \"new_individuals\" on public LB into account?
\nI feel like its a bit challenging to incorporate these properly into a validation metric.
If you do not predict new_individuals, then LB should be around 0.112 lower by default.
", "rawMarkdown": "Do you take \"new_individuals\" on public LB into account?\nI feel like its a bit challenging to incorporate these properly into a validation metric.\n\nIf you do not predict new_individuals, then LB should be around 0.112 lower by default.", "votes": null }, { "id": "1686897", "postDate": "02/12/2022 12:30:37", "content": "So I also saw that in LB the proportion of new_individuals is 0.112, so I designed my local validation set to have the same proportion roughly. By this I mean I ensured 11.2% of the validation examples were an id that did not exist in my training data, and the other examples were selected randomly.
\nI still get the same result, that there is a dramatic difference between local MAP@5 and LB. Perhaps I am not calculating the metric correctly.
", "rawMarkdown": "So I also saw that in LB the proportion of new_individuals is 0.112, so I designed my local validation set to have the same proportion roughly. By this I mean I ensured 11.2% of the validation examples were an id that did not exist in my training data, and the other examples were selected randomly.\n\nI still get the same result, that there is a dramatic difference between local MAP@5 and LB. Perhaps I am not calculating the metric correctly.", "votes": null }, { "id": "1686941", "postDate": "02/12/2022 13:04:35", "content": "Then this does not seem to be the problem.
\nKeep in mind that the new_individuals class is not randomly distributed on test set.
So I just did submissions without new_individual class to evaluate my metric, while also not considering new_individuals in my validation metric.
\nI initially also had the false implementation in the beginning.
\nI did Precision@5 instead of MAP@5. Maybe you are doing something similar? I easily got confused there.
Good luck!
", "rawMarkdown": "Then this does not seem to be the problem.\nKeep in mind that the new_individuals class is not randomly distributed on test set.\n\nSo I just did submissions without new_individual class to evaluate my metric, while also not considering new_individuals in my validation metric.\n\nI initially also had the false implementation in the beginning. \nI did Precision@5 instead of MAP@5. Maybe you are doing something similar? I easily got confused there.\n\nGood luck!", "votes": null }, { "id": "1687007", "postDate": "02/12/2022 14:29:18", "content": "Thanks for your suggestions, and good luck to you too :)
", "rawMarkdown": "Thanks for your suggestions, and good luck to you too :)", "votes": null }, { "id": "1687086", "postDate": "02/12/2022 16:03:41", "content": "@jpbremer care to say your cv score?
", "rawMarkdown": "jpbremer care to say your cv score?", "votes": null }, { "id": "1687098", "postDate": "02/12/2022 16:08:23", "content": "CV 0.8, LB 0.63
\nDont habe CV for 0.72, as its a bit more complicated.
", "rawMarkdown": "CV 0.8, LB 0.63\n\nDont habe CV for 0.72, as its a bit more complicated.", "votes": null }, { "id": "1687112", "postDate": "02/12/2022 16:14:55", "content": "How are you splitting your data into folds?
", "rawMarkdown": "How are you splitting your data into folds?", "votes": null }, { "id": "1687178", "postDate": "02/12/2022 16:51:46", "content": "wow, there's good gap between them.
\nHow are you splitting the data? I am just using startifiedKfold on individual id, not correlating with LB at all
Quick question, Do you train KNN or something similar every epoch and check CV using MAP@5 ? Just wondering.
", "rawMarkdown": "Quick question, Do you train KNN or something similar every epoch and check CV using MAP@5 ? Just wondering.", "votes": null } ], "comments": [ { "id": 1686848, "author_name": "jpbremer", "author_url": "", "post_date": "02/12/2022 11:40:09", "content": "Do you take \"new_individuals\" on public LB into account?
\nI feel like its a bit challenging to incorporate these properly into a validation metric.
If you do not predict new_individuals, then LB should be around 0.112 lower by default.
", "votes": null, "replies": [ { "id": 1686897, "author_name": "taindow", "author_url": "", "post_date": "02/12/2022 12:30:37", "content": "So I also saw that in LB the proportion of new_individuals is 0.112, so I designed my local validation set to have the same proportion roughly. By this I mean I ensured 11.2% of the validation examples were an id that did not exist in my training data, and the other examples were selected randomly.
\nI still get the same result, that there is a dramatic difference between local MAP@5 and LB. Perhaps I am not calculating the metric correctly.
", "votes": null, "replies": [] }, { "id": 1686941, "author_name": "jpbremer", "author_url": "", "post_date": "02/12/2022 13:04:35", "content": "Then this does not seem to be the problem.
\nKeep in mind that the new_individuals class is not randomly distributed on test set.
So I just did submissions without new_individual class to evaluate my metric, while also not considering new_individuals in my validation metric.
\nI initially also had the false implementation in the beginning.
\nI did Precision@5 instead of MAP@5. Maybe you are doing something similar? I easily got confused there.
Good luck!
", "votes": null, "replies": [] }, { "id": 1687007, "author_name": "taindow", "author_url": "", "post_date": "02/12/2022 14:29:18", "content": "Thanks for your suggestions, and good luck to you too :)
", "votes": null, "replies": [] }, { "id": 1687086, "author_name": "mrinath", "author_url": "", "post_date": "02/12/2022 16:03:41", "content": "@jpbremer care to say your cv score?
", "votes": null, "replies": [] }, { "id": 1687098, "author_name": "jpbremer", "author_url": "", "post_date": "02/12/2022 16:08:23", "content": "CV 0.8, LB 0.63
\nDont habe CV for 0.72, as its a bit more complicated.
", "votes": null, "replies": [] }, { "id": 1687178, "author_name": "mrinath", "author_url": "", "post_date": "02/12/2022 16:51:46", "content": "wow, there's good gap between them.
\nHow are you splitting the data? I am just using startifiedKfold on individual id, not correlating with LB at all
How are you splitting your data into folds?
", "votes": null, "replies": [] }, { "id": 1687526, "author_name": "atharvap329", "author_url": "", "post_date": "02/12/2022 22:50:19", "content": "Quick question, Do you train KNN or something similar every epoch and check CV using MAP@5 ? Just wondering.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1686707": "Hi guys, I'm just trying to get things setup so that my public LB performance tracks my local performance, but having some issues. For example, in some experiments I'm reporting local MAP@5 scores of around 0.50 on my validation set but finding it's much lower on public LB - around 0.25.\n\nIs anyone else experiencing the same result? If not, I suspect my local validation checks have a bug in there somewhere. Thanks and good luck to everyone.", "1686848": "Do you take \"new_individuals\" on public LB into account?\nI feel like its a bit challenging to incorporate these properly into a validation metric.\n\nIf you do not predict new_individuals, then LB should be around 0.112 lower by default.", "1686897": "So I also saw that in LB the proportion of new_individuals is 0.112, so I designed my local validation set to have the same proportion roughly. By this I mean I ensured 11.2% of the validation examples were an id that did not exist in my training data, and the other examples were selected randomly.\n\nI still get the same result, that there is a dramatic difference between local MAP@5 and LB. Perhaps I am not calculating the metric correctly.", "1686941": "Then this does not seem to be the problem.\nKeep in mind that the new_individuals class is not randomly distributed on test set.\n\nSo I just did submissions without new_individual class to evaluate my metric, while also not considering new_individuals in my validation metric.\n\nI initially also had the false implementation in the beginning. \nI did Precision@5 instead of MAP@5. Maybe you are doing something similar? I easily got confused there.\n\nGood luck!", "1687007": "Thanks for your suggestions, and good luck to you too :)", "1687086": "jpbremer care to say your cv score?", "1687098": "CV 0.8, LB 0.63\n\nDont habe CV for 0.72, as its a bit more complicated.", "1687112": "How are you splitting your data into folds?", "1687178": "wow, there's good gap between them.\nHow are you splitting the data? I am just using startifiedKfold on individual id, not correlating with LB at all", "1687526": "Quick question, Do you train KNN or something similar every epoch and check CV using MAP@5 ? Just wondering." }, "source": "meta" }