![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8273630%2Fa2e7046459dbbdd13738ed00d692aceb%2FHCCHurn.png?generation=1716856373232655&alt=media\")
Hello all,
\nI think many of us expected a churn here, especially after the late surge of public leaderboard scores due to the hacking elements involved. We did have a churn as expected, but probably a little bigger than gauged! Kindly find the churn visualized in a simple scattergram as below-
\n
| Region Label | \nComments | \n
|---|---|
| R1 | \n* Hearty congratulations for the stability! These participants celebrate a medal in the competition (after the post-competition sanitation) * They succeeded in staying in the medal zone in the public and private leaderboards | \n
| R2 | \n* Our team is here along with many others who heavily overfitted to the public leaderboard * I think most of us erred in the CV regime and did not trust the CV scheme enough * Perhaps some of us has some unexplored bugs in our pipelines too??? * Trusting the CV scheme and designing a trustworthy CV scheme are key elements in any competition! | \n
| R3 | \n* This and the next ones are similar * These teams beat the churn from a public LB rank close to the medal zone * Congratulations to all of these teams for the medal finish!! | \n
| R4 | \n* These teams beat the churn from a public LB rank far away from the medal zone * Congratulations to all of these teams for the medal finish!! * Staying put and not making hasty and irrational public LB based decisions is key for one to finish on a high note! | \n
| R5 | \n* We observe a few sticky regions perhaps stemming from public kernels? * These interesting regions draw attention too! | \n
Let's now look at the teams in the medal positions and analyze their local churn levels-
\n| Region Label | \nLast rank | \nComments | \n
|---|---|---|
| Gold | \n17 | \n* Stable teams = 5 * Teams entering from below = 12 | \n
| Silver | \n194 | \n* Stable teams = 49 * Teams entering from below = 119 * Teams entering from above = 9 | \n
| Bronze | \n388 | \n* Stable teams = 27 * Teams entering from below = 152 * Teams entering from above = 15 | \n
We look at the stability ratio across the medal zones to illustrate the extent of the churn. This ratio is the number of stable teams versus the medal zone size -
\n| Region Label | \nStability Ratio | \n
|---|---|
| Gold | \n29.41% | \n
| Silver | \n27.68% | \n
| Bronze | \n13.91% | \n
The reducing stability from an as-is small number in the gold zone sums up the discussion. We had an incredible churn here and an experience to remember for sure!
\nI think Kaggle competitions is a collective manifestation of human behavioral biases in a structured environment. If we track our own actions and relate them with the results obtained, we could discern a lot of herd mentality, endowment bias, myopic loss aversion and some anchoring bias components driving a lot of our sub-optimal decisions. We need to think logically and try and control our emotional biases and suppress cognitive issues to make sure we make the correct decision and ultimately land up in a position that we aspired to be at the start of the competition
\nI thank the forum contributors for their generosity, especially the early kernels that wonderfully organized the data pipeline with polars and the data-store. I am sure most of us used their work at some point in the assignment!!!
\nBest wishes for one's learning journey and hope the lessons learnt here drive better decision making in the future!!
\nAll the best and regards!
", "messages": [ { "id": "2840081", "postDate": "05/28/2024 00:57:20", "content": "Hello all,
\nI think many of us expected a churn here, especially after the late surge of public leaderboard scores due to the hacking elements involved. We did have a churn as expected, but probably a little bigger than gauged! Kindly find the churn visualized in a simple scattergram as below-
\n
| Region Label | \nComments | \n
|---|---|
| R1 | \n* Hearty congratulations for the stability! These participants celebrate a medal in the competition (after the post-competition sanitation) * They succeeded in staying in the medal zone in the public and private leaderboards | \n
| R2 | \n* Our team is here along with many others who heavily overfitted to the public leaderboard * I think most of us erred in the CV regime and did not trust the CV scheme enough * Perhaps some of us has some unexplored bugs in our pipelines too??? * Trusting the CV scheme and designing a trustworthy CV scheme are key elements in any competition! | \n
| R3 | \n* This and the next ones are similar * These teams beat the churn from a public LB rank close to the medal zone * Congratulations to all of these teams for the medal finish!! | \n
| R4 | \n* These teams beat the churn from a public LB rank far away from the medal zone * Congratulations to all of these teams for the medal finish!! * Staying put and not making hasty and irrational public LB based decisions is key for one to finish on a high note! | \n
| R5 | \n* We observe a few sticky regions perhaps stemming from public kernels? * These interesting regions draw attention too! | \n
Let's now look at the teams in the medal positions and analyze their local churn levels-
\n| Region Label | \nLast rank | \nComments | \n
|---|---|---|
| Gold | \n17 | \n* Stable teams = 5 * Teams entering from below = 12 | \n
| Silver | \n194 | \n* Stable teams = 49 * Teams entering from below = 119 * Teams entering from above = 9 | \n
| Bronze | \n388 | \n* Stable teams = 27 * Teams entering from below = 152 * Teams entering from above = 15 | \n
We look at the stability ratio across the medal zones to illustrate the extent of the churn. This ratio is the number of stable teams versus the medal zone size -
\n| Region Label | \nStability Ratio | \n
|---|---|
| Gold | \n29.41% | \n
| Silver | \n27.68% | \n
| Bronze | \n13.91% | \n
The reducing stability from an as-is small number in the gold zone sums up the discussion. We had an incredible churn here and an experience to remember for sure!
\nI think Kaggle competitions is a collective manifestation of human behavioral biases in a structured environment. If we track our own actions and relate them with the results obtained, we could discern a lot of herd mentality, endowment bias, myopic loss aversion and some anchoring bias components driving a lot of our sub-optimal decisions. We need to think logically and try and control our emotional biases and suppress cognitive issues to make sure we make the correct decision and ultimately land up in a position that we aspired to be at the start of the competition
\nI thank the forum contributors for their generosity, especially the early kernels that wonderfully organized the data pipeline with polars and the data-store. I am sure most of us used their work at some point in the assignment!!!
\nBest wishes for one's learning journey and hope the lessons learnt here drive better decision making in the future!!
\nAll the best and regards!
", "rawMarkdown": "Hello all,\n\nI think many of us expected a churn here, especially after the late surge of public leaderboard scores due to the hacking elements involved. We did have a churn as expected, but probably a little bigger than gauged! Kindly find the churn visualized in a simple scattergram as below-\n\n\n\n### Key regions of interest\n\n| Region Label | Comments |\n| --- | --- |\n| R1 | * Hearty congratulations for the stability! These participants celebrate a medal in the competition (after the post-competition sanitation)\"Do not invest too much time in competitions where CV-LB scheme is not discernable\"😂 Can't agree more
", "rawMarkdown": "\"Do not invest too much time in competitions where CV-LB scheme is not discernable\"😂 Can't agree more", "votes": null }, { "id": "2840091", "postDate": "05/28/2024 01:13:31", "content": "I finally know why so many great people don't come to this competitions 。
", "rawMarkdown": "I finally know why so many great people don't come to this competitions 。", "votes": null }, { "id": "2840095", "postDate": "05/28/2024 01:14:55", "content": "CV-LB scheme is a key element, we need to have some form of this to be able to build a model and improve it. Else, we end up gambling and relying on luck rather than on our model @f520hh
", "rawMarkdown": "CV-LB scheme is a key element, we need to have some form of this to be able to build a model and improve it. Else, we end up gambling and relying on luck rather than on our model @f520hh", "votes": null }, { "id": "2840219", "postDate": "05/28/2024 03:07:19", "content": "I second your takeaways. Little did I know 3 months ago that the best solutions would be the \"best\" because of metric hacking …
", "rawMarkdown": "I second your takeaways. Little did I know 3 months ago that the best solutions would be the \"best\" because of metric hacking ...", "votes": null }, { "id": "2840260", "postDate": "05/28/2024 03:58:51", "content": "Thanks for sharing the churn graph which makes this competition more interesting.
", "rawMarkdown": "Thanks for sharing the churn graph which makes this competition more interesting.", "votes": null }, { "id": "2841204", "postDate": "05/28/2024 12:53:42", "content": "That's great approach
", "rawMarkdown": "That's great approach", "votes": null }, { "id": "2841226", "postDate": "05/28/2024 13:18:22", "content": "Thanks for the analysis @ravi20076 ! I was in the R2 quadrant this time :(
", "rawMarkdown": "Thanks for the analysis @ravi20076 ! I was in the R2 quadrant this time :(", "votes": null }, { "id": "2841274", "postDate": "05/28/2024 13:51:37", "content": "Hey great analysis, can you explain what is cv-lb scheme?
", "rawMarkdown": "Hey great analysis, can you explain what is cv-lb scheme?", "votes": null }, { "id": "2841385", "postDate": "05/28/2024 15:05:05", "content": "Many of us are in this quadrant 😔
\nThis is the second time after ICR that such a thing happened to us!
Cross-validation results should be in sync with the leaderboard. An increase in CV score should translate in LB position improvement. Else, we are unsure what improves our models and rather rely on luck/ public leaderboard alone to make an inference @sohailmd123
", "rawMarkdown": "Cross-validation results should be in sync with the leaderboard. An increase in CV score should translate in LB position improvement. Else, we are unsure what improves our models and rather rely on luck/ public leaderboard alone to make an inference @sohailmd123", "votes": null }, { "id": "2841551", "postDate": "05/28/2024 15:58:38", "content": "Oh! thank you. Yes, that does sound logical.
", "rawMarkdown": "Oh! thank you. Yes, that does sound logical.", "votes": null }, { "id": "2842190", "postDate": "05/29/2024 00:21:13", "content": "so in this competition, anyone's cv-lb scheme is discernable?
", "rawMarkdown": "so in this competition, anyone's cv-lb scheme is discernable?", "votes": null }, { "id": "2842441", "postDate": "05/29/2024 05:14:10", "content": "Not at all @mengjingyang
", "rawMarkdown": "Not at all @mengjingyang", "votes": null }, { "id": "2842515", "postDate": "05/29/2024 06:01:01", "content": "Great work, great analysis
", "rawMarkdown": "Great work, great analysis", "votes": null }, { "id": "2843935", "postDate": "05/29/2024 19:17:18", "content": "I really enjoy your churn visualization in each competition, nice job keep it up in upcoming competition! @ravi20076
", "rawMarkdown": "I really enjoy your churn visualization in each competition, nice job keep it up in upcoming competition! @ravi20076", "votes": null }, { "id": "2846184", "postDate": "05/31/2024 02:16:50", "content": "Great, keep sharing, @ravi20076
", "rawMarkdown": "Great, keep sharing, @ravi20076", "votes": null } ], "comments": [ { "id": 2840083, "author_name": "lorryzouzelun", "author_url": "", "post_date": "05/28/2024 01:02:10", "content": "\"Do not invest too much time in competitions where CV-LB scheme is not discernable\"😂 Can't agree more
", "votes": null, "replies": [] }, { "id": 2840091, "author_name": "", "author_url": "", "post_date": "05/28/2024 01:13:31", "content": "I finally know why so many great people don't come to this competitions 。
", "votes": null, "replies": [ { "id": 2840095, "author_name": "ravi20076", "author_url": "", "post_date": "05/28/2024 01:14:55", "content": "CV-LB scheme is a key element, we need to have some form of this to be able to build a model and improve it. Else, we end up gambling and relying on luck rather than on our model @f520hh
", "votes": null, "replies": [] } ] }, { "id": 2840219, "author_name": "andreasbis", "author_url": "", "post_date": "05/28/2024 03:07:19", "content": "I second your takeaways. Little did I know 3 months ago that the best solutions would be the \"best\" because of metric hacking …
", "votes": null, "replies": [] }, { "id": 2840260, "author_name": "crsuthikshnkumar", "author_url": "", "post_date": "05/28/2024 03:58:51", "content": "Thanks for sharing the churn graph which makes this competition more interesting.
", "votes": null, "replies": [] }, { "id": 2841204, "author_name": "tomaraman", "author_url": "", "post_date": "05/28/2024 12:53:42", "content": "That's great approach
", "votes": null, "replies": [] }, { "id": 2841226, "author_name": "octaviograu", "author_url": "", "post_date": "05/28/2024 13:18:22", "content": "Thanks for the analysis @ravi20076 ! I was in the R2 quadrant this time :(
", "votes": null, "replies": [ { "id": 2841385, "author_name": "ravi20076", "author_url": "", "post_date": "05/28/2024 15:05:05", "content": "Many of us are in this quadrant 😔
\nThis is the second time after ICR that such a thing happened to us!
Hey great analysis, can you explain what is cv-lb scheme?
", "votes": null, "replies": [ { "id": 2841388, "author_name": "ravi20076", "author_url": "", "post_date": "05/28/2024 15:06:15", "content": "Cross-validation results should be in sync with the leaderboard. An increase in CV score should translate in LB position improvement. Else, we are unsure what improves our models and rather rely on luck/ public leaderboard alone to make an inference @sohailmd123
", "votes": null, "replies": [ { "id": 2841551, "author_name": "sohailmd123", "author_url": "", "post_date": "05/28/2024 15:58:38", "content": "Oh! thank you. Yes, that does sound logical.
", "votes": null, "replies": [] }, { "id": 2842190, "author_name": "mengjingyang", "author_url": "", "post_date": "05/29/2024 00:21:13", "content": "so in this competition, anyone's cv-lb scheme is discernable?
", "votes": null, "replies": [ { "id": 2842441, "author_name": "ravi20076", "author_url": "", "post_date": "05/29/2024 05:14:10", "content": "Not at all @mengjingyang
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2842515, "author_name": "divu2001", "author_url": "", "post_date": "05/29/2024 06:01:01", "content": "Great work, great analysis
", "votes": null, "replies": [] }, { "id": 2843935, "author_name": "michau96", "author_url": "", "post_date": "05/29/2024 19:17:18", "content": "I really enjoy your churn visualization in each competition, nice job keep it up in upcoming competition! @ravi20076
", "votes": null, "replies": [] }, { "id": 2846184, "author_name": "muhammedtausif", "author_url": "", "post_date": "05/31/2024 02:16:50", "content": "Great, keep sharing, @ravi20076
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2840081": "Hello all,\n\nI think many of us expected a churn here, especially after the late surge of public leaderboard scores due to the hacking elements involved. We did have a churn as expected, but probably a little bigger than gauged! Kindly find the churn visualized in a simple scattergram as below-\n\n\n\n### Key regions of interest\n\n| Region Label | Comments |\n| --- | --- |\n| R1 | * Hearty congratulations for the stability! These participants celebrate a medal in the competition (after the post-competition sanitation)