Hi, I have been stuck on getting a score of 0.002-0.005 for a while now. No matter how good my validation in training is, I can't seem to get past it. If anyone else has experienced this and managed to get through it, what did you do differently?
", "messages": [ { "id": "2577795", "postDate": "12/28/2023 20:05:16", "content": "Hi, I have been stuck on getting a score of 0.002-0.005 for a while now. No matter how good my validation in training is, I can't seem to get past it. If anyone else has experienced this and managed to get through it, what did you do differently?
", "rawMarkdown": "Hi, I have been stuck on getting a score of 0.002-0.005 for a while now. No matter how good my validation in training is, I can't seem to get past it. If anyone else has experienced this and managed to get through it, what did you do differently?", "votes": null }, { "id": "2577852", "postDate": "12/28/2023 21:08:51", "content": "Generally, if I get unreasonable scores, I try to setup the following pipeline
\n1) inference + scoring [validation]
\n2) just inference [on test]
and run two pipelines in both kaggle kernel and local env, this way you can be sure that local validation matches the code you wrote in kaggle kernel, and can be sure that inference is done correctly
", "rawMarkdown": "Generally, if I get unreasonable scores, I try to setup the following pipeline\n1) inference + scoring [validation]\n2) just inference [on test]\n\nand run two pipelines in both kaggle kernel and local env, this way you can be sure that local validation matches the code you wrote in kaggle kernel, and can be sure that inference is done correctly", "votes": null }, { "id": "2577929", "postDate": "12/28/2023 22:45:31", "content": "So by this do you mean run the inference on the same dataset i’m using for validation then also run validation on the same validation dataset and see if they line up?
", "rawMarkdown": "So by this do you mean run the inference on the same dataset i’m using for validation then also run validation on the same validation dataset and see if they line up?", "votes": null }, { "id": "2577931", "postDate": "12/28/2023 22:53:07", "content": "I try to run inference on validation in kaggle kernel and try to get the same validation score I got locally, it helps to eliminate most of the bugs
\ne.g. recently I forgot .sigmoid() and got a little bit lower score which was not expected
", "rawMarkdown": "I try to run inference on validation in kaggle kernel and try to get the same validation score I got locally, it helps to eliminate most of the bugs\n\ne.g. recently I forgot .sigmoid() and got a little bit lower score which was not expected", "votes": null }, { "id": "2577953", "postDate": "12/28/2023 23:01:50", "content": "Ah do you mainly train your models locally? I can’t train locally due to hardware limitations haha
", "rawMarkdown": "Ah do you mainly train your models locally? I can’t train locally due to hardware limitations haha", "votes": null }, { "id": "2578014", "postDate": "12/29/2023 01:17:42", "content": "You can verify your training set by examining the differences between the masks output from the training set and normal masks, to determine if there is an issue at some point in the pipeline
", "rawMarkdown": "You can verify your training set by examining the differences between the masks output from the training set and normal masks, to determine if there is an issue at some point in the pipeline", "votes": null }, { "id": "2578278", "postDate": "12/29/2023 06:09:24", "content": "You were spot on. Turns out our model was wacky and not as good as we thought it was lol. Thank you for the tip
", "rawMarkdown": "You were spot on. Turns out our model was wacky and not as good as we thought it was lol. Thank you for the tip", "votes": null }, { "id": "2580967", "postDate": "12/31/2023 12:40:48", "content": "Have you plotted some of the predicted masks? That's one way to see if you have a bug in your evaluation pipeline. Indeed, sometimes you think you have a good metric score but in reality it is either ill-implemented or you are overfitting so visual debugging helps. Best of luck!
", "rawMarkdown": "Have you plotted some of the predicted masks? That's one way to see if you have a bug in your evaluation pipeline. Indeed, sometimes you think you have a good metric score but in reality it is either ill-implemented or you are overfitting so visual debugging helps. Best of luck!", "votes": null }, { "id": "2581452", "postDate": "12/31/2023 17:46:41", "content": "I ended up completely switching my whole setup and it fixed everything. I’m now getting fair LB scores like 0.3. All that’s left now is to mess with different hyper parameters
", "rawMarkdown": "I ended up completely switching my whole setup and it fixed everything. I’m now getting fair LB scores like 0.3. All that’s left now is to mess with different hyper parameters", "votes": null } ], "comments": [ { "id": 2577852, "author_name": "martynoveduard", "author_url": "", "post_date": "12/28/2023 21:08:51", "content": "Generally, if I get unreasonable scores, I try to setup the following pipeline
\n1) inference + scoring [validation]
\n2) just inference [on test]
and run two pipelines in both kaggle kernel and local env, this way you can be sure that local validation matches the code you wrote in kaggle kernel, and can be sure that inference is done correctly
", "votes": null, "replies": [ { "id": 2577929, "author_name": "zuni8789798", "author_url": "", "post_date": "12/28/2023 22:45:31", "content": "So by this do you mean run the inference on the same dataset i’m using for validation then also run validation on the same validation dataset and see if they line up?
", "votes": null, "replies": [ { "id": 2577931, "author_name": "martynoveduard", "author_url": "", "post_date": "12/28/2023 22:53:07", "content": "I try to run inference on validation in kaggle kernel and try to get the same validation score I got locally, it helps to eliminate most of the bugs
\ne.g. recently I forgot .sigmoid() and got a little bit lower score which was not expected
", "votes": null, "replies": [ { "id": 2577953, "author_name": "zuni8789798", "author_url": "", "post_date": "12/28/2023 23:01:50", "content": "Ah do you mainly train your models locally? I can’t train locally due to hardware limitations haha
", "votes": null, "replies": [] } ] } ] }, { "id": 2578278, "author_name": "zuni8789798", "author_url": "", "post_date": "12/29/2023 06:09:24", "content": "You were spot on. Turns out our model was wacky and not as good as we thought it was lol. Thank you for the tip
", "votes": null, "replies": [] } ] }, { "id": 2578014, "author_name": "koalas", "author_url": "", "post_date": "12/29/2023 01:17:42", "content": "You can verify your training set by examining the differences between the masks output from the training set and normal masks, to determine if there is an issue at some point in the pipeline
", "votes": null, "replies": [ { "id": 2581452, "author_name": "zuni8789798", "author_url": "", "post_date": "12/31/2023 17:46:41", "content": "I ended up completely switching my whole setup and it fixed everything. I’m now getting fair LB scores like 0.3. All that’s left now is to mess with different hyper parameters
", "votes": null, "replies": [] } ] }, { "id": 2580967, "author_name": "yassinealouini", "author_url": "", "post_date": "12/31/2023 12:40:48", "content": "Have you plotted some of the predicted masks? That's one way to see if you have a bug in your evaluation pipeline. Indeed, sometimes you think you have a good metric score but in reality it is either ill-implemented or you are overfitting so visual debugging helps. Best of luck!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2577795": "Hi, I have been stuck on getting a score of 0.002-0.005 for a while now. No matter how good my validation in training is, I can't seem to get past it. If anyone else has experienced this and managed to get through it, what did you do differently?", "2577852": "Generally, if I get unreasonable scores, I try to setup the following pipeline\n1) inference + scoring [validation]\n2) just inference [on test]\n\nand run two pipelines in both kaggle kernel and local env, this way you can be sure that local validation matches the code you wrote in kaggle kernel, and can be sure that inference is done correctly", "2577929": "So by this do you mean run the inference on the same dataset i’m using for validation then also run validation on the same validation dataset and see if they line up?", "2577931": "I try to run inference on validation in kaggle kernel and try to get the same validation score I got locally, it helps to eliminate most of the bugs\n\ne.g. recently I forgot .sigmoid() and got a little bit lower score which was not expected", "2577953": "Ah do you mainly train your models locally? I can’t train locally due to hardware limitations haha", "2578014": "You can verify your training set by examining the differences between the masks output from the training set and normal masks, to determine if there is an issue at some point in the pipeline", "2578278": "You were spot on. Turns out our model was wacky and not as good as we thought it was lol. Thank you for the tip", "2580967": "Have you plotted some of the predicted masks? That's one way to see if you have a bug in your evaluation pipeline. Indeed, sometimes you think you have a good metric score but in reality it is either ill-implemented or you are overfitting so visual debugging helps. Best of luck!", "2581452": "I ended up completely switching my whole setup and it fixed everything. I’m now getting fair LB scores like 0.3. All that’s left now is to mess with different hyper parameters" }, "source": "meta" }