Congratulations to the winners and everyone that participated. I also want to thank the organizers for giving us a chance to explore this very challenging data.
\n\nNow since only 6 teams that ranked between 1st to 25th places on the public LB ended up within 25 places on the private LB. This means majority of us may have overfitted to the public LB.
\n\nMy observations like many others is the difficulty of getting consistent results with NNs. It was really hard to get a good cross validation setup for this problem. So I am posing the question to the community to answer on hindsight what went wrong?
\n\nMy best model on hindsight is a CatBoost model with public LB = 0.648, private LB=0.655, and CV scores 0.719 ± 0.094 but I did not choose it as one of my final submissions. This would have put me in bronze medal at number 80 position.
\n\nPlease share you opinions below as a way to brainstorm whilst we wait for the solutions of the top performers.
", "messages": [ { "id": "496169", "postDate": "03/22/2019 00:26:08", "content": "Congratulations to the winners and everyone that participated. I also want to thank the organizers for giving us a chance to explore this very challenging data.
\n\nNow since only 6 teams that ranked between 1st to 25th places on the public LB ended up within 25 places on the private LB. This means majority of us may have overfitted to the public LB.
\n\nMy observations like many others is the difficulty of getting consistent results with NNs. It was really hard to get a good cross validation setup for this problem. So I am posing the question to the community to answer on hindsight what went wrong?
\n\nMy best model on hindsight is a CatBoost model with public LB = 0.648, private LB=0.655, and CV scores 0.719 ± 0.094 but I did not choose it as one of my final submissions. This would have put me in bronze medal at number 80 position.
\n\nPlease share you opinions below as a way to brainstorm whilst we wait for the solutions of the top performers.
", "rawMarkdown": "Congratulations to the winners and everyone that participated. I also want to thank the organizers for giving us a chance to explore this very challenging data.\n\nNow since only 6 teams that ranked between 1st to 25th places on the public LB ended up within 25 places on the private LB. This means majority of us may have overfitted to the public LB. \n\nMy observations like many others is the difficulty of getting consistent results with NNs. It was really hard to get a good cross validation setup for this problem. So I am posing the question to the community to answer on hindsight what went wrong?\n\nMy best model on hindsight is a [CatBoost model](https://www.kaggle.com/c/vsb-power-line-fault-detection/discussion/85146) with **public LB = 0.648, private LB=0.655, and CV scores 0.719 ± 0.094** but I did not choose it as one of my final submissions. This would have put me in bronze medal at number 80 position.\n\nPlease share you opinions below as a way to brainstorm whilst we wait for the solutions of the top performers.", "votes": null }, { "id": "496226", "postDate": "03/22/2019 01:36:25", "content": "I was working on normal LGBM (CV: 0.72, public LB 0.612 private 0.650).\nI should have stop there, but it just looked too low !\nI blend it with the LSTM 0.7 solution and bye bye medal :(
\n\nOn another note, the LSTM using denoised signal fare nicely (private LB 0.656, public 0.632). It was the 5 fold LSTM kernel with adding just hpf and wavelet denoising. Sadly I didn't thought of choosing it in the last day :(
\n\nWhat went wrong.....poor decision making on my side. I shall go and get myself drunk this weekend. TGIF !
", "rawMarkdown": "I was working on normal LGBM (CV: 0.72, public LB 0.612 private 0.650).\nI should have stop there, but it just looked too low !\nI blend it with the LSTM 0.7 solution and bye bye medal :( \n\nOn another note, the LSTM using denoised signal fare nicely (private LB 0.656, public 0.632). It was the 5 fold LSTM kernel with adding just hpf and wavelet denoising. Sadly I didn't thought of choosing it in the last day :( \n\nWhat went wrong.....poor decision making on my side. I shall go and get myself drunk this weekend. TGIF !", "votes": null }, { "id": "496234", "postDate": "03/22/2019 01:46:18", "content": "Thanks for sharing @nyleve. You and me both but go easy on the beer, you need to maintain this brilliant mind of yours :-)
", "rawMarkdown": "Thanks for sharing @nyleve. You and me both but go easy on the beer, you need to maintain this brilliant mind of yours :-)", "votes": null }, { "id": "496285", "postDate": "03/22/2019 02:55:17", "content": "Hi YaGana, a good experience for both of us and many ... I am still thinking about the proper validation strategy in a situation like this, but also have another question in mind.
\n\nWould you mind sharing your thought on the issue here:\nhttps://www.kaggle.com/c/vsb-power-line-fault-detection/discussion/85167
", "rawMarkdown": "Hi YaGana, a good experience for both of us and many ... I am still thinking about the proper validation strategy in a situation like this, but also have another question in mind.\n\nWould you mind sharing your thought on the issue here:\nhttps://www.kaggle.com/c/vsb-power-line-fault-detection/discussion/85167", "votes": null }, { "id": "496338", "postDate": "03/22/2019 04:04:53", "content": "@ratthachat, I added my comments there.
", "rawMarkdown": "ratthachat, I added my comments there.", "votes": null }, { "id": "496890", "postDate": "03/22/2019 17:58:02", "content": "Very similar experience. 0.666 on private LB would have gotten the silver. LSTM with wavelet denoising, but the public LB was so low. It is especially painful coming right after Microsoft Malware. Funny both these challenges had about 60% data in public LB, unlike other vision challenges. Folks who stayed with LightGBM seem to have won out. Max Halford publicly gave up, released the code and then ended up winning. Happy for him :)
", "rawMarkdown": "Very similar experience. 0.666 on private LB would have gotten the silver. LSTM with wavelet denoising, but the public LB was so low. It is especially painful coming right after Microsoft Malware. Funny both these challenges had about 60% data in public LB, unlike other vision challenges. Folks who stayed with LightGBM seem to have won out. Max Halford publicly gave up, released the code and then ended up winning. Happy for him :)", "votes": null }, { "id": "497026", "postDate": "03/22/2019 21:31:40", "content": "Thanks fo sharing @liger1776. I did not realize that is the guy that won.
\n\nFunny, I did a well tuned LGBM with the same features as my catboost model during the competition and the numbers are not good i.e. public LB = 0.558, private LB = 0.566
", "rawMarkdown": "Thanks fo sharing @liger1776. I did not realize that is the guy that won. \n\nFunny, I did a well tuned LGBM with the same features as my catboost model during the competition and the numbers are not good i.e. public LB = 0.558, private LB = 0.566", "votes": null } ], "comments": [ { "id": 496226, "author_name": "nyleve", "author_url": "", "post_date": "03/22/2019 01:36:25", "content": "I was working on normal LGBM (CV: 0.72, public LB 0.612 private 0.650).\nI should have stop there, but it just looked too low !\nI blend it with the LSTM 0.7 solution and bye bye medal :(
\n\nOn another note, the LSTM using denoised signal fare nicely (private LB 0.656, public 0.632). It was the 5 fold LSTM kernel with adding just hpf and wavelet denoising. Sadly I didn't thought of choosing it in the last day :(
\n\nWhat went wrong.....poor decision making on my side. I shall go and get myself drunk this weekend. TGIF !
", "votes": null, "replies": [ { "id": 496234, "author_name": "sheriytm", "author_url": "", "post_date": "03/22/2019 01:46:18", "content": "Thanks for sharing @nyleve. You and me both but go easy on the beer, you need to maintain this brilliant mind of yours :-)
", "votes": null, "replies": [] }, { "id": 496890, "author_name": "liger1776", "author_url": "", "post_date": "03/22/2019 17:58:02", "content": "Very similar experience. 0.666 on private LB would have gotten the silver. LSTM with wavelet denoising, but the public LB was so low. It is especially painful coming right after Microsoft Malware. Funny both these challenges had about 60% data in public LB, unlike other vision challenges. Folks who stayed with LightGBM seem to have won out. Max Halford publicly gave up, released the code and then ended up winning. Happy for him :)
", "votes": null, "replies": [] }, { "id": 497026, "author_name": "sheriytm", "author_url": "", "post_date": "03/22/2019 21:31:40", "content": "Thanks fo sharing @liger1776. I did not realize that is the guy that won.
\n\nFunny, I did a well tuned LGBM with the same features as my catboost model during the competition and the numbers are not good i.e. public LB = 0.558, private LB = 0.566
", "votes": null, "replies": [] } ] }, { "id": 496285, "author_name": "ratthachat", "author_url": "", "post_date": "03/22/2019 02:55:17", "content": "Hi YaGana, a good experience for both of us and many ... I am still thinking about the proper validation strategy in a situation like this, but also have another question in mind.
\n\nWould you mind sharing your thought on the issue here:\nhttps://www.kaggle.com/c/vsb-power-line-fault-detection/discussion/85167
", "votes": null, "replies": [ { "id": 496338, "author_name": "sheriytm", "author_url": "", "post_date": "03/22/2019 04:04:53", "content": "@ratthachat, I added my comments there.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "496169": "Congratulations to the winners and everyone that participated. I also want to thank the organizers for giving us a chance to explore this very challenging data.\n\nNow since only 6 teams that ranked between 1st to 25th places on the public LB ended up within 25 places on the private LB. This means majority of us may have overfitted to the public LB. \n\nMy observations like many others is the difficulty of getting consistent results with NNs. It was really hard to get a good cross validation setup for this problem. So I am posing the question to the community to answer on hindsight what went wrong?\n\nMy best model on hindsight is a [CatBoost model](https://www.kaggle.com/c/vsb-power-line-fault-detection/discussion/85146) with **public LB = 0.648, private LB=0.655, and CV scores 0.719 ± 0.094** but I did not choose it as one of my final submissions. This would have put me in bronze medal at number 80 position.\n\nPlease share you opinions below as a way to brainstorm whilst we wait for the solutions of the top performers.", "496226": "I was working on normal LGBM (CV: 0.72, public LB 0.612 private 0.650).\nI should have stop there, but it just looked too low !\nI blend it with the LSTM 0.7 solution and bye bye medal :( \n\nOn another note, the LSTM using denoised signal fare nicely (private LB 0.656, public 0.632). It was the 5 fold LSTM kernel with adding just hpf and wavelet denoising. Sadly I didn't thought of choosing it in the last day :( \n\nWhat went wrong.....poor decision making on my side. I shall go and get myself drunk this weekend. TGIF !", "496234": "Thanks for sharing @nyleve. You and me both but go easy on the beer, you need to maintain this brilliant mind of yours :-)", "496285": "Hi YaGana, a good experience for both of us and many ... I am still thinking about the proper validation strategy in a situation like this, but also have another question in mind.\n\nWould you mind sharing your thought on the issue here:\nhttps://www.kaggle.com/c/vsb-power-line-fault-detection/discussion/85167", "496338": "ratthachat, I added my comments there.", "496890": "Very similar experience. 0.666 on private LB would have gotten the silver. LSTM with wavelet denoising, but the public LB was so low. It is especially painful coming right after Microsoft Malware. Funny both these challenges had about 60% data in public LB, unlike other vision challenges. Folks who stayed with LightGBM seem to have won out. Max Halford publicly gave up, released the code and then ended up winning. Happy for him :)", "497026": "Thanks fo sharing @liger1776. I did not realize that is the guy that won. \n\nFunny, I did a well tuned LGBM with the same features as my catboost model during the competition and the numbers are not good i.e. public LB = 0.558, private LB = 0.566" }, "source": "meta" }