![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8273630%2F23d44d80e8321341acf030955efc6dfd%2Flb.png?generation=1738819119448147&alt=media\")
Hello all,
\nThis competition ended a few hours ago as on writing this post with a relatively stable leaderboard and low amount of churn. Let's delve into this as below-
\n
We have 2 regions of interest as below-
\n| Region label | \nComments and inferences | \n
|---|---|
| R1 | \n* This is the medal winning region, with all gold, silver and bronze medals aggregated into 1 single group * It is evident that we have low churn here and public and private leaderboard are stable for this group | \n
| R2 | \n* This represents a small section of the participant base that churned down on the private leaderboard from a moderate public leaderboard position, perhaps due to some code issues, a public kernel, blending issues, etc. * This is a small group though and the leaderboard otherwise looks stable for the entire participant base | \n
This section analyses the ranking churn and submissions across the gold, silver and bronze medal winners as below-
\n| Medal zone | \nLast rank | \nComments | \nAverage submissions | \n
|---|---|---|---|
| Gold | \n11 | \n* Stable teams = 10 * Entering from below = 1 | \n222.1 | \n
| Silver | \n50 | \n* Stable teams = 37 * Entering from below = 1 * Entering from above = 1 | \n155.10 | \n
| Bronze | \n100 | \n* Stable teams = 46 * Entering from below = 3 * Entering from above = 1 | \n110.06 | \n
It is very evident from the above table that we have an incredible level of stability across the public and private leaderboards. Also, on average, top 10 teams submitted 70-75 times more than the rest of the silver medal winners. Also, bronze medal winners on average submitted 40 counts lesser than the silver rankers and almost half of the gold positions!
Wishing one and all the best for their future endeavors, wishing the winners hearty congratulations and happy learning to one and all!
\nBest regards,
\nRavi Ramakrishnan
Hello all,
\nThis competition ended a few hours ago as on writing this post with a relatively stable leaderboard and low amount of churn. Let's delve into this as below-
\n
We have 2 regions of interest as below-
\n| Region label | \nComments and inferences | \n
|---|---|
| R1 | \n* This is the medal winning region, with all gold, silver and bronze medals aggregated into 1 single group * It is evident that we have low churn here and public and private leaderboard are stable for this group | \n
| R2 | \n* This represents a small section of the participant base that churned down on the private leaderboard from a moderate public leaderboard position, perhaps due to some code issues, a public kernel, blending issues, etc. * This is a small group though and the leaderboard otherwise looks stable for the entire participant base | \n
This section analyses the ranking churn and submissions across the gold, silver and bronze medal winners as below-
\n| Medal zone | \nLast rank | \nComments | \nAverage submissions | \n
|---|---|---|---|
| Gold | \n11 | \n* Stable teams = 10 * Entering from below = 1 | \n222.1 | \n
| Silver | \n50 | \n* Stable teams = 37 * Entering from below = 1 * Entering from above = 1 | \n155.10 | \n
| Bronze | \n100 | \n* Stable teams = 46 * Entering from below = 3 * Entering from above = 1 | \n110.06 | \n
It is very evident from the above table that we have an incredible level of stability across the public and private leaderboards. Also, on average, top 10 teams submitted 70-75 times more than the rest of the silver medal winners. Also, bronze medal winners on average submitted 40 counts lesser than the silver rankers and almost half of the gold positions!
Wishing one and all the best for their future endeavors, wishing the winners hearty congratulations and happy learning to one and all!
\nBest regards,
\nRavi Ramakrishnan
Thank you for the excellent analysis.
\nI am one of the medalists with significant rank changes.
\nPublic: 73rd → Private: 90th
I have been ensembling three models and adjusting the softmax probabilities and cluster size thresholds of each model to improve CV and Public scores.
\nSub1 Public 0.737, Private 0.721
\nSub2 Public 0.729, Private 0.719
I submitted two final submissions. Sub1 had the best CV and Public scores, but a slight change in the softmax or cluster size drastically lowered the CV and scores. Sub2's scores did not change much regardless of changes in the softmax or cluster size. As a result, Sub1 was the best in Private, but the rank dropped because it overfitted to the Public dataset. However, considering that the learning data consisted of 7 sets and the Public evaluation was based on 130 sets of data, I expected the rank wouldn't drop much even if it overfitted to the Public dataset.
\nI've only been participating in Kaggle for a few months, but I think I might have won a medal, and I'm thrilled.
", "rawMarkdown": "Thank you for the excellent analysis.\n\nI am one of the medalists with significant rank changes. \nPublic: 73rd → Private: 90th\n\nI have been ensembling three models and adjusting the softmax probabilities and cluster size thresholds of each model to improve CV and Public scores.\n\nSub1 Public 0.737, Private 0.721\nSub2 Public 0.729, Private 0.719\n\nI submitted two final submissions. Sub1 had the best CV and Public scores, but a slight change in the softmax or cluster size drastically lowered the CV and scores. Sub2's scores did not change much regardless of changes in the softmax or cluster size. As a result, Sub1 was the best in Private, but the rank dropped because it overfitted to the Public dataset. However, considering that the learning data consisted of 7 sets and the Public evaluation was based on 130 sets of data, I expected the rank wouldn't drop much even if it overfitted to the Public dataset.\n\nI've only been participating in Kaggle for a few months, but I think I might have won a medal, and I'm thrilled.", "votes": null }, { "id": "3116901", "postDate": "02/06/2025 12:24:13", "content": "From the solutions I read, I see that everyone was having consistent results between public and private as you have mentioned. Some people solely relied on public LB as well by probing.
", "rawMarkdown": "From the solutions I read, I see that everyone was having consistent results between public and private as you have mentioned. Some people solely relied on public LB as well by probing.", "votes": null }, { "id": "3116906", "postDate": "02/06/2025 12:27:15", "content": "True - this is a risky strategy (I will think 10 times before doing this) but it worked for them @snnclsr
", "rawMarkdown": "True - this is a risky strategy (I will think 10 times before doing this) but it worked for them @snnclsr", "votes": null }, { "id": "3116911", "postDate": "02/06/2025 12:30:38", "content": "Yeah, it's true. I think because of the dataset paper as well. People were more confident specific to this competition.
", "rawMarkdown": "Yeah, it's true. I think because of the dataset paper as well. People were more confident specific to this competition.", "votes": null }, { "id": "3117053", "postDate": "02/06/2025 15:34:27", "content": "This is incredible, thank you so much for this analysis.
", "rawMarkdown": "This is incredible, thank you so much for this analysis.", "votes": null }, { "id": "3118356", "postDate": "02/07/2025 23:09:02", "content": "Remarkable work @ravi20076 - gives a clearer perspective on both Public and Private LB performance, with a better understanding of performance clustering for R1 and R2. I wish o other competitions could adopt this as a post-mortem after each competition.
", "rawMarkdown": "Remarkable work @ravi20076 - gives a clearer perspective on both Public and Private LB performance, with a better understanding of performance clustering for R1 and R2. I wish o other competitions could adopt this as a post-mortem after each competition.", "votes": null }, { "id": "3118460", "postDate": "02/08/2025 05:35:59", "content": "I learn a lot from this analysis - the leaderboard here was stable and cv-lb was stable @olabodejames
\nSometimes, the leaderboard goes awry and people lose out on the private leaderboard due to a churn and data issues between the public and private leaderboard.
I usually stay away from churn-ridden competitions / invest less time on them - there is no point of investing time on a risky assignment where result is not certain!
", "rawMarkdown": "I learn a lot from this analysis - the leaderboard here was stable and cv-lb was stable @olabodejames \nSometimes, the leaderboard goes awry and people lose out on the private leaderboard due to a churn and data issues between the public and private leaderboard.\n\nI usually stay away from churn-ridden competitions / invest less time on them - there is no point of investing time on a risky assignment where result is not certain!", "votes": null }, { "id": "3118944", "postDate": "02/08/2025 17:15:30", "content": "Absolutely, you are apt on churn-ridden competitions as those are susceptible to very high uncertainty which results in efforts misdirections - which I have seen a lot of
", "rawMarkdown": "Absolutely, you are apt on churn-ridden competitions as those are susceptible to very high uncertainty which results in efforts misdirections - which I have seen a lot of", "votes": null } ], "comments": [ { "id": 3116696, "author_name": "ryqsaq", "author_url": "", "post_date": "02/06/2025 08:17:03", "content": "Thank you for the excellent analysis.
\nI am one of the medalists with significant rank changes.
\nPublic: 73rd → Private: 90th
I have been ensembling three models and adjusting the softmax probabilities and cluster size thresholds of each model to improve CV and Public scores.
\nSub1 Public 0.737, Private 0.721
\nSub2 Public 0.729, Private 0.719
I submitted two final submissions. Sub1 had the best CV and Public scores, but a slight change in the softmax or cluster size drastically lowered the CV and scores. Sub2's scores did not change much regardless of changes in the softmax or cluster size. As a result, Sub1 was the best in Private, but the rank dropped because it overfitted to the Public dataset. However, considering that the learning data consisted of 7 sets and the Public evaluation was based on 130 sets of data, I expected the rank wouldn't drop much even if it overfitted to the Public dataset.
\nI've only been participating in Kaggle for a few months, but I think I might have won a medal, and I'm thrilled.
", "votes": null, "replies": [] }, { "id": 3116901, "author_name": "snnclsr", "author_url": "", "post_date": "02/06/2025 12:24:13", "content": "From the solutions I read, I see that everyone was having consistent results between public and private as you have mentioned. Some people solely relied on public LB as well by probing.
", "votes": null, "replies": [ { "id": 3116906, "author_name": "ravi20076", "author_url": "", "post_date": "02/06/2025 12:27:15", "content": "True - this is a risky strategy (I will think 10 times before doing this) but it worked for them @snnclsr
", "votes": null, "replies": [ { "id": 3116911, "author_name": "snnclsr", "author_url": "", "post_date": "02/06/2025 12:30:38", "content": "Yeah, it's true. I think because of the dataset paper as well. People were more confident specific to this competition.
", "votes": null, "replies": [] } ] } ] }, { "id": 3117053, "author_name": "rezaparaan", "author_url": "", "post_date": "02/06/2025 15:34:27", "content": "This is incredible, thank you so much for this analysis.
", "votes": null, "replies": [] }, { "id": 3118356, "author_name": "olabodejames", "author_url": "", "post_date": "02/07/2025 23:09:02", "content": "Remarkable work @ravi20076 - gives a clearer perspective on both Public and Private LB performance, with a better understanding of performance clustering for R1 and R2. I wish o other competitions could adopt this as a post-mortem after each competition.
", "votes": null, "replies": [ { "id": 3118460, "author_name": "ravi20076", "author_url": "", "post_date": "02/08/2025 05:35:59", "content": "I learn a lot from this analysis - the leaderboard here was stable and cv-lb was stable @olabodejames
\nSometimes, the leaderboard goes awry and people lose out on the private leaderboard due to a churn and data issues between the public and private leaderboard.
I usually stay away from churn-ridden competitions / invest less time on them - there is no point of investing time on a risky assignment where result is not certain!
", "votes": null, "replies": [ { "id": 3118944, "author_name": "olabodejames", "author_url": "", "post_date": "02/08/2025 17:15:30", "content": "Absolutely, you are apt on churn-ridden competitions as those are susceptible to very high uncertainty which results in efforts misdirections - which I have seen a lot of
", "votes": null, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "3116591": "Hello all,\n\nThis competition ended a few hours ago as on writing this post with a relatively stable leaderboard and low amount of churn. Let's delve into this as below-\n\n\n\n## Regions of interest\n\nWe have 2 regions of interest as below-\n| Region label | Comments and inferences|\n| --- | --- |\n| R1| * This is the medal winning region, with all gold, silver and bronze medals aggregated into 1 single group