I am trying to repeat the top score kernel using Pytorch, and only achieves score less than 0.6. It seems that in this competition, the model performance fluctuate a lot, which is abnormal for other competition. I am wondering if someone has a solution to the unstable performance?
", "messages": [ { "id": "489787", "postDate": "03/14/2019 04:30:12", "content": "I am trying to repeat the top score kernel using Pytorch, and only achieves score less than 0.6. It seems that in this competition, the model performance fluctuate a lot, which is abnormal for other competition. I am wondering if someone has a solution to the unstable performance?
", "rawMarkdown": "I am trying to repeat the top score kernel using Pytorch, and only achieves score less than 0.6. It seems that in this competition, the model performance fluctuate a lot, which is abnormal for other competition. I am wondering if someone has a solution to the unstable performance?", "votes": null }, { "id": "489943", "postDate": "03/14/2019 07:28:24", "content": "Yes, this is one of the critical factor! We can expect some shake-up in this competition, like Malware Competition. Huge shake-up is so brutal! :D. I still have challenge on formulating a very good validation strategy that can withstand changes on both Public and Private LB.
", "rawMarkdown": "Yes, this is one of the critical factor! We can expect some shake-up in this competition, like Malware Competition. Huge shake-up is so brutal! :D. I still have challenge on formulating a very good validation strategy that can withstand changes on both Public and Private LB.", "votes": null }, { "id": "490180", "postDate": "03/14/2019 10:53:02", "content": "Could ensembling or stacking help in this regard ? I am yet to try this.
", "rawMarkdown": "Could ensembling or stacking help in this regard ? I am yet to try this.", "votes": null }, { "id": "490703", "postDate": "03/14/2019 19:15:23", "content": "Waiting for your update!
", "rawMarkdown": "Waiting for your update!", "votes": null }, { "id": "490704", "postDate": "03/14/2019 19:15:49", "content": "May I ask what is your validation score for your best single model now?
", "rawMarkdown": "May I ask what is your validation score for your best single model now?", "votes": null }, { "id": "491678", "postDate": "03/16/2019 01:30:32", "content": "Same here. I too expect a huge shake-up in the results given this. I'm also eager to know how others have handled this issue.
", "rawMarkdown": "Same here. I too expect a huge shake-up in the results given this. I'm also eager to know how others have handled this issue.", "votes": null }, { "id": "491686", "postDate": "03/16/2019 01:48:05", "content": "my validation score for my current LB score is 0.718084. But I'm not trusting either my validation & LB score simply because my validation strategy are not robust enough.
", "rawMarkdown": "my validation score for my current LB score is 0.718084. But I'm not trusting either my validation & LB score simply because my validation strategy are not robust enough.", "votes": null }, { "id": "491744", "postDate": "03/16/2019 04:34:32", "content": "At least your validation score is close to your LB score.
", "rawMarkdown": "At least your validation score is close to your LB score.", "votes": null }, { "id": "494010", "postDate": "03/19/2019 11:11:07", "content": "@strideradu I tried ensembling and the public LB score was barely better than the worst model in the ensemble. :(
", "rawMarkdown": "strideradu I tried ensembling and the public LB score was barely better than the worst model in the ensemble. :(", "votes": null }, { "id": "494067", "postDate": "03/19/2019 12:36:00", "content": "For me, majority voting leads to the almost same or less scored result
", "rawMarkdown": "For me, majority voting leads to the almost same or less scored result", "votes": null }, { "id": "494296", "postDate": "03/19/2019 16:53:40", "content": "Looks like we need to find a way to make one strong model with robust validation. This seems to be better than ensembling, majority voting or stacking for this particular competition.
", "rawMarkdown": "Looks like we need to find a way to make one strong model with robust validation. This seems to be better than ensembling, majority voting or stacking for this particular competition.", "votes": null } ], "comments": [ { "id": 489943, "author_name": "", "author_url": "", "post_date": "03/14/2019 07:28:24", "content": "Yes, this is one of the critical factor! We can expect some shake-up in this competition, like Malware Competition. Huge shake-up is so brutal! :D. I still have challenge on formulating a very good validation strategy that can withstand changes on both Public and Private LB.
", "votes": null, "replies": [ { "id": 490704, "author_name": "strideradu", "author_url": "", "post_date": "03/14/2019 19:15:49", "content": "May I ask what is your validation score for your best single model now?
", "votes": null, "replies": [] }, { "id": 491686, "author_name": "", "author_url": "", "post_date": "03/16/2019 01:48:05", "content": "my validation score for my current LB score is 0.718084. But I'm not trusting either my validation & LB score simply because my validation strategy are not robust enough.
", "votes": null, "replies": [] }, { "id": 491744, "author_name": "behrad3d", "author_url": "", "post_date": "03/16/2019 04:34:32", "content": "At least your validation score is close to your LB score.
", "votes": null, "replies": [] } ] }, { "id": 490180, "author_name": "tarunpaparaju", "author_url": "", "post_date": "03/14/2019 10:53:02", "content": "Could ensembling or stacking help in this regard ? I am yet to try this.
", "votes": null, "replies": [ { "id": 490703, "author_name": "strideradu", "author_url": "", "post_date": "03/14/2019 19:15:23", "content": "Waiting for your update!
", "votes": null, "replies": [] }, { "id": 494010, "author_name": "tarunpaparaju", "author_url": "", "post_date": "03/19/2019 11:11:07", "content": "@strideradu I tried ensembling and the public LB score was barely better than the worst model in the ensemble. :(
", "votes": null, "replies": [] }, { "id": 494067, "author_name": "soonhwankwon", "author_url": "", "post_date": "03/19/2019 12:36:00", "content": "For me, majority voting leads to the almost same or less scored result
", "votes": null, "replies": [] }, { "id": 494296, "author_name": "tarunpaparaju", "author_url": "", "post_date": "03/19/2019 16:53:40", "content": "Looks like we need to find a way to make one strong model with robust validation. This seems to be better than ensembling, majority voting or stacking for this particular competition.
", "votes": null, "replies": [] } ] }, { "id": 491678, "author_name": "behrad3d", "author_url": "", "post_date": "03/16/2019 01:30:32", "content": "Same here. I too expect a huge shake-up in the results given this. I'm also eager to know how others have handled this issue.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "489787": "I am trying to repeat the top score kernel using Pytorch, and only achieves score less than 0.6. It seems that in this competition, the model performance fluctuate a lot, which is abnormal for other competition. I am wondering if someone has a solution to the unstable performance?", "489943": "Yes, this is one of the critical factor! We can expect some shake-up in this competition, like Malware Competition. Huge shake-up is so brutal! :D. I still have challenge on formulating a very good validation strategy that can withstand changes on both Public and Private LB.", "490180": "Could ensembling or stacking help in this regard ? I am yet to try this.", "490703": "Waiting for your update!", "490704": "May I ask what is your validation score for your best single model now?", "491678": "Same here. I too expect a huge shake-up in the results given this. I'm also eager to know how others have handled this issue.", "491686": "my validation score for my current LB score is 0.718084. But I'm not trusting either my validation & LB score simply because my validation strategy are not robust enough.", "491744": "At least your validation score is close to your LB score.", "494010": "strideradu I tried ensembling and the public LB score was barely better than the worst model in the ensemble. :(", "494067": "For me, majority voting leads to the almost same or less scored result", "494296": "Looks like we need to find a way to make one strong model with robust validation. This seems to be better than ensembling, majority voting or stacking for this particular competition." }, "source": "meta" }