Playing with hyperparams my notebooks Val score increases like crazy but seems in overfiting and this doesn't generalize to lb
\nhttps://www.kaggle.com/code/jainam213/0-97val0-12lb
\nWe need a more robust validation dataset to correlate cv and lb closely
Playing with hyperparams my notebooks Val score increases like crazy but seems in overfiting and this doesn't generalize to lb
\nhttps://www.kaggle.com/code/jainam213/0-97val0-12lb
\nWe need a more robust validation dataset to correlate cv and lb closely
Thank you for pointing this out. I skimmed through your notebook, and it looks like you use the final evaluation metric for validation score. The top-k score for the tile collection is definitely not suitable for validation because your model can just get lucky and happen to pick one good config in the top k candidates and have high validation score. In our experiments, we use different validation metrics. In particular, we're using MSE or MAPE for tile collection, and OPA (https://www.tensorflow.org/ranking/api_docs/python/tfr/keras/metrics/OPAMetric) for layout collections. You may want to try those.
", "rawMarkdown": "Thank you for pointing this out. I skimmed through your notebook, and it looks like you use the final evaluation metric for validation score. The top-k score for the tile collection is definitely not suitable for validation because your model can just get lucky and happen to pick one good config in the top k candidates and have high validation score. In our experiments, we use different validation metrics. In particular, we're using MSE or MAPE for tile collection, and OPA (https://www.tensorflow.org/ranking/api_docs/python/tfr/keras/metrics/OPAMetric) for layout collections. You may want to try those.", "votes": null }, { "id": "2419144", "postDate": "09/01/2023 17:58:57", "content": "Oooh Thanks a ton
", "rawMarkdown": "Oooh Thanks a ton", "votes": null } ], "comments": [ { "id": 2419099, "author_name": "mangpophothilimthana", "author_url": "", "post_date": "09/01/2023 17:22:12", "content": "Thank you for pointing this out. I skimmed through your notebook, and it looks like you use the final evaluation metric for validation score. The top-k score for the tile collection is definitely not suitable for validation because your model can just get lucky and happen to pick one good config in the top k candidates and have high validation score. In our experiments, we use different validation metrics. In particular, we're using MSE or MAPE for tile collection, and OPA (https://www.tensorflow.org/ranking/api_docs/python/tfr/keras/metrics/OPAMetric) for layout collections. You may want to try those.
", "votes": null, "replies": [ { "id": 2419144, "author_name": "jainam213", "author_url": "", "post_date": "09/01/2023 17:58:57", "content": "Oooh Thanks a ton
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2418136": "Playing with hyperparams my notebooks Val score increases like crazy but seems in overfiting and this doesn't generalize to lb\nhttps://www.kaggle.com/code/jainam213/0-97val0-12lb\nWe need a more robust validation dataset to correlate cv and lb closely", "2419099": "Thank you for pointing this out. I skimmed through your notebook, and it looks like you use the final evaluation metric for validation score. The top-k score for the tile collection is definitely not suitable for validation because your model can just get lucky and happen to pick one good config in the top k candidates and have high validation score. In our experiments, we use different validation metrics. In particular, we're using MSE or MAPE for tile collection, and OPA (https://www.tensorflow.org/ranking/api_docs/python/tfr/keras/metrics/OPAMetric) for layout collections. You may want to try those.", "2419144": "Oooh Thanks a ton" }, "source": "meta" }