{ "id": 478602, "title": "[0.34] LB Gatekeeping", "url": "/competitions/hms-harmful-brain-activity-classification/discussion/478602", "author_name": "", "post_date": "2024-02-21T13:59:04.518120100Z", "votes": 61, "comment_count": 19, "views": 0, "content": "

Hi,

\n

Having seen a rapid public leaderboard improvement over last week, especially, due to the public high scoring notebooks, I'd like to raise a warning for those who are new to Kaggle. Some of the high scoring notebooks are to be taken with caution.

\n

Let's look at 0.34 LB example, and see what is going on by comparing versions:

\n

\n

Well, we can see arbitrary assigned weights to weighted average ensemble. It is not based on the second level model with oofs or anything of this nature, it just a pure gambling for 3 seconds of fame and 100 forks without upvotes.

\n

If you are really up to ensembling/blending your/public models, do it properly, please. Spend some time, sweat a bit and research on Chris Delotte, Giba, or any other publicly available methods. Otherwise, the value of your published notebook is Null, less or equal to this topic.

\n

Peace!

", "messages": [ { "id": "2661785", "postDate": "02/21/2024 13:59:04", "content": "

Hi,

\n

Having seen a rapid public leaderboard improvement over last week, especially, due to the public high scoring notebooks, I'd like to raise a warning for those who are new to Kaggle. Some of the high scoring notebooks are to be taken with caution.

\n

Let's look at 0.34 LB example, and see what is going on by comparing versions:

\n

\n

Well, we can see arbitrary assigned weights to weighted average ensemble. It is not based on the second level model with oofs or anything of this nature, it just a pure gambling for 3 seconds of fame and 100 forks without upvotes.

\n

If you are really up to ensembling/blending your/public models, do it properly, please. Spend some time, sweat a bit and research on Chris Delotte, Giba, or any other publicly available methods. Otherwise, the value of your published notebook is Null, less or equal to this topic.

\n

Peace!

", "rawMarkdown": "Hi,\n\nHaving seen a rapid public leaderboard improvement over last week, especially, due to the public high scoring notebooks, I'd like to raise a warning for those who are new to Kaggle. Some of the high scoring notebooks are to be taken **with caution**. \n\nLet's look at **0.34 LB example**, and see what is going on by comparing versions:\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6259210%2F6c32a88edc224b2b65de547972693333%2FGatekeeping.png?generation=1708523044874225&alt=media)\n\nWell, we can see arbitrary assigned weights to weighted average ensemble. It is not based on the second level model with oofs or anything of this nature, it just a pure gambling for 3 seconds of fame and 100 forks without upvotes.\n\nIf you are really up to ensembling/blending your/public models, do it properly, please. Spend some time, sweat a bit and research on Chris Delotte, Giba, or any other publicly available methods. Otherwise, the value of your published notebook is Null, less or equal to this topic. \n\nPeace!", "votes": null }, { "id": "2661889", "postDate": "02/21/2024 15:07:14", "content": "

Looks like blendfest started early in this competition

", "rawMarkdown": "Looks like blendfest started early in this competition", "votes": null }, { "id": "2661891", "postDate": "02/21/2024 15:14:23", "content": "

It will be really hard for someone with actually good models to find a teammate among 0.34- LB…

", "rawMarkdown": "It will be really hard for someone with actually good models to find a teammate among 0.34- LB...", "votes": null }, { "id": "2661950", "postDate": "02/21/2024 15:53:21", "content": "

@gunesevitan 🤣🤣

\n

", "rawMarkdown": "gunesevitan 🤣🤣 \n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6259210%2F6a4c76b2f1fea1a004aacfab2b60a82e%2Fpublic%20probing.png?generation=1708531552967583&alt=media)", "votes": null }, { "id": "2661952", "postDate": "02/21/2024 15:55:52", "content": "

@samson8 I did not think about it, it is a good point… I believe someone can solve it by explicitly mentioning \"looking for teammates with solo model 0.3 - 0.34 or so\".

", "rawMarkdown": "samson8 I did not think about it, it is a good point... I believe someone can solve it by explicitly mentioning \"looking for teammates with solo model 0.3 - 0.34 or so\".", "votes": null }, { "id": "2661983", "postDate": "02/21/2024 16:18:54", "content": "

it just a pure gambling for 3 seconds of fame and 100 forks without upvotes

\n

Brutal.

", "rawMarkdown": "**`it just a pure gambling for 3 seconds of fame and 100 forks without upvotes`**\n\nBrutal.", "votes": null }, { "id": "2661994", "postDate": "02/21/2024 16:26:42", "content": "

@sergiosaharovskiy

\n

", "rawMarkdown": "sergiosaharovskiy \n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3197853%2F4d911be8a3de685265213c2ff85fc214%2FScreenshot%202024-02-21%20at%2013.25.39.png?generation=1708532834738776&alt=media)", "votes": null }, { "id": "2662044", "postDate": "02/21/2024 17:05:08", "content": "

It seems that 0.36 is a large proportion in that notebook, I think it overfits LB

", "rawMarkdown": "It seems that 0.36 is a large proportion in that notebook, I think it overfits LB", "votes": null }, { "id": "2662544", "postDate": "02/22/2024 01:05:09", "content": "

There are too many edge cases in training datasets. If these datasets are hidden in private test datasets, it's dangerous. Actually, the public score is too optimistic for us. It reminds me of Sennet.

", "rawMarkdown": "There are too many edge cases in training datasets. If these datasets are hidden in private test datasets, it's dangerous. Actually, the public score is too optimistic for us. It reminds me of Sennet.", "votes": null }, { "id": "2662577", "postDate": "02/22/2024 02:16:10", "content": "

I totally agree with you. But seeing high scores on the leaderboard is so satisfying… 😂

", "rawMarkdown": "I totally agree with you. But seeing high scores on the leaderboard is so satisfying... 😂", "votes": null }, { "id": "2662590", "postDate": "02/22/2024 02:28:10", "content": "

But perhaps can't we think about LB weighting as a kind of ensembling parameter finding with public test datasets with limited trials?

", "rawMarkdown": "But perhaps can't we think about LB weighting as a kind of ensembling parameter finding with public test datasets with limited trials?", "votes": null }, { "id": "2662609", "postDate": "02/22/2024 02:51:13", "content": "

Of course we can, but all of the models in this notebook come from models with high loss (over 0.4).

", "rawMarkdown": "Of course we can, but all of the models in this notebook come from models with high loss (over 0.4).", "votes": null }, { "id": "2663502", "postDate": "02/22/2024 14:20:05", "content": "

Someone posted yesterday public notebook titled \"4 Model Ensemble (LB: 0.34) - Version 1\". It was deleted later. Unfortunately I didn't catch the author's name. I've copied and submitted while it was alive. Probably someone also did the same. Feeling myself uncomfortable now because the notebook removal. Plus in the title I see 'copied from private notebook'. Hope it will not be considered as private sharing later.

", "rawMarkdown": "Someone posted yesterday public notebook titled \"**4 Model Ensemble (LB: 0.34) - Version 1\"**. It was deleted later. Unfortunately I didn't catch the author's name. I've copied and submitted while it was alive. Probably someone also did the same. Feeling myself uncomfortable now because the notebook removal. Plus in the title I see 'copied from private notebook'. Hope it will not be considered as private sharing later.", "votes": null }, { "id": "2663634", "postDate": "02/22/2024 15:55:40", "content": "

Agree .. Not sure I am the only one but when we tried based on OOF I see very good cv but not LB . Also I see my keras models have excellent CV but not LB unlike pyTorch models .

", "rawMarkdown": "Agree .. Not sure I am the only one but when we tried based on OOF I see very good cv but not LB . Also I see my keras models have excellent CV but not LB unlike pyTorch models .", "votes": null }, { "id": "2663657", "postDate": "02/22/2024 16:11:15", "content": "

@samvelkoch no worries. It is was not the private sharing. It was publicly available for anyone even though for a short time.

", "rawMarkdown": "samvelkoch no worries. It is was not the private sharing. It was publicly available for anyone even though for a short time.", "votes": null }, { "id": "2664217", "postDate": "02/22/2024 21:39:46", "content": "

My current LB score (0.34) is an ensemble, but it is largely based on a single 5-fold model that has a LB score of 0.34…so there is hope!

", "rawMarkdown": "My current LB score (0.34) is an ensemble, but it is largely based on a single 5-fold model that has a LB score of 0.34...so there is hope!", "votes": null }, { "id": "2664464", "postDate": "02/23/2024 03:19:29", "content": "

it is good

", "rawMarkdown": "it is good", "votes": null }, { "id": "2667961", "postDate": "02/25/2024 12:52:24", "content": "

Super Agree with you.
\nI completely trust CV.

", "rawMarkdown": "Super Agree with you.\nI completely trust CV.", "votes": null }, { "id": "2671551", "postDate": "02/27/2024 15:09:59", "content": "

It's difficult to focus on studying and researching consistently and seeing these ensembles dominate the LB at the same time and let it pass, wait and not pay attention to position in the LB….
\nI believe that in the long run makes a big difference when it comes to the right time to do the ensemble blending.

", "rawMarkdown": "It's difficult to focus on studying and researching consistently and seeing these ensembles dominate the LB at the same time and let it pass, wait and not pay attention to position in the LB....\nI believe that in the long run makes a big difference when it comes to the right time to do the ensemble blending.", "votes": null }, { "id": "2671672", "postDate": "02/27/2024 16:34:50", "content": "

agree with you

", "rawMarkdown": "agree with you", "votes": null } ], "comments": [ { "id": 2661889, "author_name": "gunesevitan", "author_url": "", "post_date": "02/21/2024 15:07:14", "content": "

Looks like blendfest started early in this competition

", "votes": null, "replies": [ { "id": 2661950, "author_name": "sergiosaharovskiy", "author_url": "", "post_date": "02/21/2024 15:53:21", "content": "

@gunesevitan 🤣🤣

\n

", "votes": null, "replies": [ { "id": 2661994, "author_name": "alejopaullier", "author_url": "", "post_date": "02/21/2024 16:26:42", "content": "

@sergiosaharovskiy

\n

", "votes": null, "replies": [] } ] } ] }, { "id": 2661891, "author_name": "samson8", "author_url": "", "post_date": "02/21/2024 15:14:23", "content": "

It will be really hard for someone with actually good models to find a teammate among 0.34- LB…

", "votes": null, "replies": [ { "id": 2661952, "author_name": "sergiosaharovskiy", "author_url": "", "post_date": "02/21/2024 15:55:52", "content": "

@samson8 I did not think about it, it is a good point… I believe someone can solve it by explicitly mentioning \"looking for teammates with solo model 0.3 - 0.34 or so\".

", "votes": null, "replies": [] } ] }, { "id": 2661983, "author_name": "yantxx", "author_url": "", "post_date": "02/21/2024 16:18:54", "content": "

it just a pure gambling for 3 seconds of fame and 100 forks without upvotes

\n

Brutal.

", "votes": null, "replies": [] }, { "id": 2662044, "author_name": "gentlezdh", "author_url": "", "post_date": "02/21/2024 17:05:08", "content": "

It seems that 0.36 is a large proportion in that notebook, I think it overfits LB

", "votes": null, "replies": [] }, { "id": 2662544, "author_name": "sweetyheehee", "author_url": "", "post_date": "02/22/2024 01:05:09", "content": "

There are too many edge cases in training datasets. If these datasets are hidden in private test datasets, it's dangerous. Actually, the public score is too optimistic for us. It reminds me of Sennet.

", "votes": null, "replies": [] }, { "id": 2662577, "author_name": "junseonglee11", "author_url": "", "post_date": "02/22/2024 02:16:10", "content": "

I totally agree with you. But seeing high scores on the leaderboard is so satisfying… 😂

", "votes": null, "replies": [ { "id": 2662590, "author_name": "junseonglee11", "author_url": "", "post_date": "02/22/2024 02:28:10", "content": "

But perhaps can't we think about LB weighting as a kind of ensembling parameter finding with public test datasets with limited trials?

", "votes": null, "replies": [ { "id": 2662609, "author_name": "sweetyheehee", "author_url": "", "post_date": "02/22/2024 02:51:13", "content": "

Of course we can, but all of the models in this notebook come from models with high loss (over 0.4).

", "votes": null, "replies": [] } ] } ] }, { "id": 2663502, "author_name": "samvelkoch", "author_url": "", "post_date": "02/22/2024 14:20:05", "content": "

Someone posted yesterday public notebook titled \"4 Model Ensemble (LB: 0.34) - Version 1\". It was deleted later. Unfortunately I didn't catch the author's name. I've copied and submitted while it was alive. Probably someone also did the same. Feeling myself uncomfortable now because the notebook removal. Plus in the title I see 'copied from private notebook'. Hope it will not be considered as private sharing later.

", "votes": null, "replies": [ { "id": 2663657, "author_name": "sergiosaharovskiy", "author_url": "", "post_date": "02/22/2024 16:11:15", "content": "

@samvelkoch no worries. It is was not the private sharing. It was publicly available for anyone even though for a short time.

", "votes": null, "replies": [] } ] }, { "id": 2663634, "author_name": "gauravbrills", "author_url": "", "post_date": "02/22/2024 15:55:40", "content": "

Agree .. Not sure I am the only one but when we tried based on OOF I see very good cv but not LB . Also I see my keras models have excellent CV but not LB unlike pyTorch models .

", "votes": null, "replies": [] }, { "id": 2664217, "author_name": "seanbearden", "author_url": "", "post_date": "02/22/2024 21:39:46", "content": "

My current LB score (0.34) is an ensemble, but it is largely based on a single 5-fold model that has a LB score of 0.34…so there is hope!

", "votes": null, "replies": [] }, { "id": 2664464, "author_name": "monicquinn", "author_url": "", "post_date": "02/23/2024 03:19:29", "content": "

it is good

", "votes": null, "replies": [] }, { "id": 2667961, "author_name": "haruki741", "author_url": "", "post_date": "02/25/2024 12:52:24", "content": "

Super Agree with you.
\nI completely trust CV.

", "votes": null, "replies": [] }, { "id": 2671551, "author_name": "rafaelzimmermann1", "author_url": "", "post_date": "02/27/2024 15:09:59", "content": "

It's difficult to focus on studying and researching consistently and seeing these ensembles dominate the LB at the same time and let it pass, wait and not pay attention to position in the LB….
\nI believe that in the long run makes a big difference when it comes to the right time to do the ensemble blending.

", "votes": null, "replies": [] }, { "id": 2671672, "author_name": "saionaverma", "author_url": "", "post_date": "02/27/2024 16:34:50", "content": "

agree with you

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2661785": "Hi,\n\nHaving seen a rapid public leaderboard improvement over last week, especially, due to the public high scoring notebooks, I'd like to raise a warning for those who are new to Kaggle. Some of the high scoring notebooks are to be taken **with caution**. \n\nLet's look at **0.34 LB example**, and see what is going on by comparing versions:\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6259210%2F6c32a88edc224b2b65de547972693333%2FGatekeeping.png?generation=1708523044874225&alt=media)\n\nWell, we can see arbitrary assigned weights to weighted average ensemble. It is not based on the second level model with oofs or anything of this nature, it just a pure gambling for 3 seconds of fame and 100 forks without upvotes.\n\nIf you are really up to ensembling/blending your/public models, do it properly, please. Spend some time, sweat a bit and research on Chris Delotte, Giba, or any other publicly available methods. Otherwise, the value of your published notebook is Null, less or equal to this topic. \n\nPeace!", "2661889": "Looks like blendfest started early in this competition", "2661891": "It will be really hard for someone with actually good models to find a teammate among 0.34- LB...", "2661950": "gunesevitan 🤣🤣 \n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6259210%2F6a4c76b2f1fea1a004aacfab2b60a82e%2Fpublic%20probing.png?generation=1708531552967583&alt=media)", "2661952": "samson8 I did not think about it, it is a good point... I believe someone can solve it by explicitly mentioning \"looking for teammates with solo model 0.3 - 0.34 or so\".", "2661983": "**`it just a pure gambling for 3 seconds of fame and 100 forks without upvotes`**\n\nBrutal.", "2661994": "sergiosaharovskiy \n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3197853%2F4d911be8a3de685265213c2ff85fc214%2FScreenshot%202024-02-21%20at%2013.25.39.png?generation=1708532834738776&alt=media)", "2662044": "It seems that 0.36 is a large proportion in that notebook, I think it overfits LB", "2662544": "There are too many edge cases in training datasets. If these datasets are hidden in private test datasets, it's dangerous. Actually, the public score is too optimistic for us. It reminds me of Sennet.", "2662577": "I totally agree with you. But seeing high scores on the leaderboard is so satisfying... 😂", "2662590": "But perhaps can't we think about LB weighting as a kind of ensembling parameter finding with public test datasets with limited trials?", "2662609": "Of course we can, but all of the models in this notebook come from models with high loss (over 0.4).", "2663502": "Someone posted yesterday public notebook titled \"**4 Model Ensemble (LB: 0.34) - Version 1\"**. It was deleted later. Unfortunately I didn't catch the author's name. I've copied and submitted while it was alive. Probably someone also did the same. Feeling myself uncomfortable now because the notebook removal. Plus in the title I see 'copied from private notebook'. Hope it will not be considered as private sharing later.", "2663634": "Agree .. Not sure I am the only one but when we tried based on OOF I see very good cv but not LB . Also I see my keras models have excellent CV but not LB unlike pyTorch models .", "2663657": "samvelkoch no worries. It is was not the private sharing. It was publicly available for anyone even though for a short time.", "2664217": "My current LB score (0.34) is an ensemble, but it is largely based on a single 5-fold model that has a LB score of 0.34...so there is hope!", "2664464": "it is good", "2667961": "Super Agree with you.\nI completely trust CV.", "2671551": "It's difficult to focus on studying and researching consistently and seeing these ensembles dominate the LB at the same time and let it pass, wait and not pay attention to position in the LB....\nI believe that in the long run makes a big difference when it comes to the right time to do the ensemble blending.", "2671672": "agree with you" }, "source": "meta" }