\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3268512%2F6a55b4ba0ac7360846cb279af7e141f1%2F12-26_1.png?generation=1609005085844642&alt=media\")
I'm pretty new to image recognition competitions, so forgive me if I'm missing something.
\nI trained a model that got pretty high validation AUC (I'm pretty sure there was some train-validation leakage since I trained the model in multiple sessions with shuffled train/validation datasets).
\n
Even though there might be some leakage, my leaderboard score got a 0.500, which is really confusing. I would expect my model to perform better, even if there was train/validation leakage. I ran it again with a different way to load the TFRecords (no changes to training the model) and it scored a 0.507. Furthermore, the predictions look really normal; for the most part, the model is pretty confident. I've included a snip of predictions below if that's allowed.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3268512%2Fde3fcc9d5d3de185b9afc297133f86a6%2F12-26_2.png?generation=1609005655328816&alt=media\")
I'm really confused on what's going on. Help or ideas on what could be causing this discrepancy is much appreciated. Thanks!
", "messages": [ { "id": "1127651", "postDate": "12/26/2020 18:01:28", "content": "I'm pretty new to image recognition competitions, so forgive me if I'm missing something.
\nI trained a model that got pretty high validation AUC (I'm pretty sure there was some train-validation leakage since I trained the model in multiple sessions with shuffled train/validation datasets).
\n
Even though there might be some leakage, my leaderboard score got a 0.500, which is really confusing. I would expect my model to perform better, even if there was train/validation leakage. I ran it again with a different way to load the TFRecords (no changes to training the model) and it scored a 0.507. Furthermore, the predictions look really normal; for the most part, the model is pretty confident. I've included a snip of predictions below if that's allowed.
\n
I'm really confused on what's going on. Help or ideas on what could be causing this discrepancy is much appreciated. Thanks!
", "rawMarkdown": "I'm pretty new to image recognition competitions, so forgive me if I'm missing something.\n\nI trained a model that got pretty high validation AUC (I'm pretty sure there was some train-validation leakage since I trained the model in multiple sessions with shuffled train/validation datasets).\n\n\nEven though there might be some leakage, my leaderboard score got a 0.500, which is really confusing. I would expect my model to perform better, even if there was train/validation leakage. I ran it again with a different way to load the TFRecords (no changes to training the model) and it scored a 0.507. Furthermore, the predictions look really normal; for the most part, the model is pretty confident. I've included a snip of predictions below if that's allowed.\n\n\nI'm really confused on what's going on. Help or ideas on what could be causing this discrepancy is much appreciated. Thanks!", "votes": null }, { "id": "1127796", "postDate": "12/26/2020 21:49:04", "content": "Make sure you don't have a shuffle in your inference. You are getting random results.
\nIf still confused, run the inference against the train data and make sure you get something reasonable.
", "rawMarkdown": "Make sure you don't have a shuffle in your inference. You are getting random results. \n\nIf still confused, run the inference against the train data and make sure you get something reasonable.", "votes": null } ], "comments": [ { "id": 1127796, "author_name": "richardepstein", "author_url": "", "post_date": "12/26/2020 21:49:04", "content": "Make sure you don't have a shuffle in your inference. You are getting random results.
\nIf still confused, run the inference against the train data and make sure you get something reasonable.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1127651": "I'm pretty new to image recognition competitions, so forgive me if I'm missing something.\n\nI trained a model that got pretty high validation AUC (I'm pretty sure there was some train-validation leakage since I trained the model in multiple sessions with shuffled train/validation datasets).\n\n\nEven though there might be some leakage, my leaderboard score got a 0.500, which is really confusing. I would expect my model to perform better, even if there was train/validation leakage. I ran it again with a different way to load the TFRecords (no changes to training the model) and it scored a 0.507. Furthermore, the predictions look really normal; for the most part, the model is pretty confident. I've included a snip of predictions below if that's allowed.\n\n\nI'm really confused on what's going on. Help or ideas on what could be causing this discrepancy is much appreciated. Thanks!", "1127796": "Make sure you don't have a shuffle in your inference. You are getting random results. \n\nIf still confused, run the inference against the train data and make sure you get something reasonable." }, "source": "meta" }