![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F347463%2F6dd8331b2feb74047b42dff6d424b87d%2Funderstanding_cloud_organization-shakeup.png?generation=1574121627698353&alt=media\")
Congratulations to all medalists and more generally all who got better at iterating and improving their solutions along the way :)
\n\nHere's a shake-up scatter plot for Understanding Clouds from Satellite Images - this shows public vs private leaderboard ranks (top/left are drops and bottom/right are rises upwards).
\n\n
Perhaps some day Kaggle will build this into their UI for competitions, but until that day arrives you can find them all generated in a kernel over in the Meta Kaggle dataset that creates shake-up scatter plots for nearly all competitions. (See also this one for tables.) Check them out to see how this competition compares...
\n\nThe stats are:\n - shake-up = 0.07132\n - top 10% shake-up = 0.02128\n - gold shake-up = 0.00762
\n\nBiggest fall: 977 places (I won't point anyone out)
\n\nBest rise was DatNT, gaining 714 places.
\n\nEdit: Here's a top 10 leaper league table!
\n\n@ Newer Kagglers - the shake-up originator is BreakfastPirate... Check out his discussions tab for many similar shake-up reports/threads and some great \"choose wisely\" memes.
", "messages": [ { "id": "676050", "postDate": "11/18/2019 23:59:41", "content": "Congratulations to all medalists and more generally all who got better at iterating and improving their solutions along the way :)
\n\nHere's a shake-up scatter plot for Understanding Clouds from Satellite Images - this shows public vs private leaderboard ranks (top/left are drops and bottom/right are rises upwards).
\n\n
Perhaps some day Kaggle will build this into their UI for competitions, but until that day arrives you can find them all generated in a kernel over in the Meta Kaggle dataset that creates shake-up scatter plots for nearly all competitions. (See also this one for tables.) Check them out to see how this competition compares...
\n\nThe stats are:\n - shake-up = 0.07132\n - top 10% shake-up = 0.02128\n - gold shake-up = 0.00762
\n\nBiggest fall: 977 places (I won't point anyone out)
\n\nBest rise was DatNT, gaining 714 places.
\n\nEdit: Here's a top 10 leaper league table!
\n\n@ Newer Kagglers - the shake-up originator is BreakfastPirate... Check out his discussions tab for many similar shake-up reports/threads and some great \"choose wisely\" memes.
", "rawMarkdown": "Congratulations to all medalists and more generally all who got better at iterating and improving their solutions along the way :)\n\nHere's a shake-up scatter plot for *Understanding Clouds from Satellite Images* - this shows public vs private leaderboard ranks (top/left are drops and bottom/right are rises *upwards*).\n\n\n\nPerhaps some day Kaggle will build this into their UI for competitions, but until that day arrives you can find them all generated in [a kernel][1] over in the [Meta Kaggle dataset][3] that creates shake-up scatter plots for nearly all competitions. (See also [this one][2] for tables.) Check them out to see how this competition compares...\n\nThe stats are:\n - shake-up = 0.07132\n - top 10% shake-up = 0.02128\n - gold shake-up = 0.00762\n\nBiggest fall: 977 places (I won't point anyone out)\n\nBest rise was **DatNT**, gaining 714 places.\n\nEdit: Here's a top 10 leaper league table!\n\n - *+714* [1055 → 341] DatNT (@moewie94 )\n - *+688* [884 → 196] Cakey (@cakeisgood )\n - *+629* [870 → 241] Vitaly Agapitov (@datomi )\n - *+621* [844 → 223] Dzianis Sivets (@dsivets )\n - *+620* [856 → 236] CatEek (@cateek )\n - *+619* [906 → 287] Sizhe Li (@seaself )\n - *+615* [833 → 218] Brandon (@bhenrym14 )\n - *+609* [842 → 233] King (@zcy111222 )\n - *+608* [852 → 244] hics33 (@hics33 )\n - *+604* [882 → 278] Kenal Butani (@kenal99 @mesh19 @anjali100 )\n\n@ Newer Kagglers - the shake-up originator is [BreakfastPirate][4]... Check out his [discussions tab][4] for many similar shake-up reports/threads and some great \"choose wisely\" memes.\n\n [1]: https://www.kaggle.com/jtrotman/meta-kaggle-scatter-plot-competition-shake-up\n [2]: https://www.kaggle.com/jtrotman/meta-kaggle-competition-shake-up\n [3]: https://www.kaggle.com/kaggle/meta-kaggle\n [4]: https://www.kaggle.com/breakfastpirate/discussion\n [5]: https://www.youtube.com/watch?v=A0TalLrtZ24&t=0m51s", "votes": null }, { "id": "676133", "postDate": "11/19/2019 01:33:19", "content": "Head and Tail is still .. its the tummy that shook !
", "rawMarkdown": "Head and Tail is still .. its the tummy that shook !", "votes": null }, { "id": "676196", "postDate": "11/19/2019 02:36:55", "content": "Thanks. I suggest adding histogram/density plot of shake-up differences.
", "rawMarkdown": "Thanks. I suggest adding histogram/density plot of shake-up differences.", "votes": null }, { "id": "676621", "postDate": "11/19/2019 11:09:54", "content": "Yes, that would look good on a score vs score plot ... but this plot is rank vs rank, so the histogram of each axis will simply be uniform.
\n\nThere is another kernel that shows the line traces of how the scores change going up the leaderboard for each comp - I'll run it when this comp enters into Meta Kaggle, but here is a preview (just private LB scores):
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F347463%2F93391dabff98f7ad82159d6405d89ada%2Funderstanding_cloud_organization-scores.png?generation=1574161726199039&alt=media\")
@jtrotman : Thanks for the wonderful visualization. But would love to see an edit on who jumped the most as well :). It will give an idea on how trusting your CV score can improve your rank and overfitting Public LB can drop your rank.
", "rawMarkdown": "jtrotman : Thanks for the wonderful visualization. But would love to see an edit on who jumped the most as well :). It will give an idea on how trusting your CV score can improve your rank and overfitting Public LB can drop your rank.", "votes": null }, { "id": "677005", "postDate": "11/19/2019 17:47:38", "content": "Thanks @manojprabhaakr - I added a mini league table to the main post...
\n\nFor investigating CV versus LB performance there is a wealth of information in Meta Kaggle - all teams public/private submission scores are in there, for nearly all competitions. (Usually takes 1-2 days to add competitions so this one should be there by the weekend.)
\n\nI made this Notebook to look at the progression over time of public/private scores for the winning team of each comp - it can help you see which types of competitions the winners make steady and reliable progress in, compared to other competitions which look noisier...
\n\nOr you could try forking this one and setting your own username at the top to see your teams progress over time :) Congrats on the bronze top 100 finish!
", "rawMarkdown": "Thanks @manojprabhaakr - I added a mini league table to the main post...\n\nFor investigating CV versus LB performance there is a wealth of information in [Meta Kaggle][1] - all teams public/private submission scores are in there, for nearly all competitions. (Usually takes 1-2 days to add competitions so this one should be there by the weekend.)\n\nI made [this Notebook][3] to look at the progression over time of public/private scores for the winning team of each comp - it can help you see which types of competitions the winners make steady and reliable progress in, compared to other competitions which look noisier...\n\nOr you could [try forking this one][2] and setting your own username at the top to see your teams progress over time :) Congrats on the bronze top 100 finish!\n\n [1]: https://www.kaggle.com/kaggle/meta-kaggle\n [2]: https://www.kaggle.com/jtrotman/user-competition-submission-traces\n [3]: https://www.kaggle.com/jtrotman/winning-team-submission-traces", "votes": null }, { "id": "677451", "postDate": "11/20/2019 07:03:50", "content": "Thanks for the amazing visualization
", "rawMarkdown": "Thanks for the amazing visualization", "votes": null }, { "id": "677836", "postDate": "11/20/2019 16:34:03", "content": "Thanks for your wishes :).
\n\nSure @jtrotman . Will do some analysis on this during this weekend. Trusting Public LB vs CV and share one more kernel. :).
", "rawMarkdown": "Thanks for your wishes :). \n\nSure @jtrotman . Will do some analysis on this during this weekend. Trusting Public LB vs CV and share one more kernel. :).", "votes": null }, { "id": "678156", "postDate": "11/21/2019 03:18:34", "content": "Easy for understanding \nI really love Scatter Plot > <\nThank you for Visualization
", "rawMarkdown": "Easy for understanding \nI really love Scatter Plot > <\nThank you for Visualization", "votes": null }, { "id": "678428", "postDate": "11/21/2019 11:34:25", "content": "I noticed some points clustering, are those the results of the same public kernel with different hyperparameters?
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3383009%2F58fb83f2a577c639850c4679da07a831%2Fshapeupplot.PNG?generation=1574335974748484&alt=media\")
Yes - exactly that - there are always a lot of people simply resubmitting public Notebooks, hoping for a quick/easy bronze I guess, or maybe as a 'backup' submission, which their other selection fails to beat.
\n\nThe main block seems to be 130 teams with private score 0.64626, from 534 - 663. They all dropped down into those positions. Tied scores are supposed to be sorted by time so there may be some difference in public/private sorting.
\n\nWith 5dp scores there could be accidental collisions, but Meta Kaggle has full precision scores in the Submissions.csv - you can detect true copies that way. I have some code that counts numbers of 'cheap' submissions per user (public and private scores seen more than once) - some have so many they are clearly auto-scraping/polling the kernel listings and submitting the best kernels automatically (and blends of the best) - in a chase for ranking points! Feels like veering into the taking-it-all-too-seriously to publish it though ;)
\n\nSubmitting public submissions does occasionally win medals, the best (\"worst?\") example is the recent Quora code competition - there's a huge streak of over 200 identical private scores starting in bronze & ending outside it - all from one shared Notebook. (I wonder if Kaggle even ran it for each team or just duplicated the result?!)
", "rawMarkdown": "Yes - exactly that - there are always a lot of people simply resubmitting public Notebooks, hoping for a quick/easy bronze I guess, or maybe as a 'backup' submission, which their other selection fails to beat.\n\nThe main block seems to be 130 teams with private score 0.64626, from 534 - 663. They all dropped down into those positions. Tied scores are supposed to be sorted by time so there may be some difference in public/private sorting.\n\nWith 5dp scores there *could* be accidental collisions, but [Meta Kaggle][2] has full precision scores in the Submissions.csv - you can detect true copies that way. I have some code that counts numbers of 'cheap' submissions per user (public and private scores seen more than once) - some have so many they are clearly auto-scraping/polling the kernel listings and submitting the best kernels automatically (and blends of the best) - in a chase for ranking points! Feels like veering into the *taking-it-all-too-seriously* to publish it though ;)\n\nSubmitting public submissions does occasionally win medals, the best (\"worst?\") example is the recent [Quora code competition][1] - there's a huge streak of over 200 identical private scores starting in bronze & ending outside it - all from one shared Notebook. (I wonder if Kaggle even ran it for each team or just duplicated the result?!) \n\n\n [1]: https://www.kaggle.com/jtrotman/meta-kaggle-scatter-plot-competition-shake-up#quora-insincere-questions-classification\n [2]: https://www.kaggle.com/kaggle/meta-kaggle", "votes": null } ], "comments": [ { "id": 676133, "author_name": "phoenix9032", "author_url": "", "post_date": "11/19/2019 01:33:19", "content": "Head and Tail is still .. its the tummy that shook !
", "votes": null, "replies": [] }, { "id": 676196, "author_name": "niuddd", "author_url": "", "post_date": "11/19/2019 02:36:55", "content": "Thanks. I suggest adding histogram/density plot of shake-up differences.
", "votes": null, "replies": [ { "id": 676621, "author_name": "jtrotman", "author_url": "", "post_date": "11/19/2019 11:09:54", "content": "Yes, that would look good on a score vs score plot ... but this plot is rank vs rank, so the histogram of each axis will simply be uniform.
\n\nThere is another kernel that shows the line traces of how the scores change going up the leaderboard for each comp - I'll run it when this comp enters into Meta Kaggle, but here is a preview (just private LB scores):
\n\n
@jtrotman : Thanks for the wonderful visualization. But would love to see an edit on who jumped the most as well :). It will give an idea on how trusting your CV score can improve your rank and overfitting Public LB can drop your rank.
", "votes": null, "replies": [ { "id": 677005, "author_name": "jtrotman", "author_url": "", "post_date": "11/19/2019 17:47:38", "content": "Thanks @manojprabhaakr - I added a mini league table to the main post...
\n\nFor investigating CV versus LB performance there is a wealth of information in Meta Kaggle - all teams public/private submission scores are in there, for nearly all competitions. (Usually takes 1-2 days to add competitions so this one should be there by the weekend.)
\n\nI made this Notebook to look at the progression over time of public/private scores for the winning team of each comp - it can help you see which types of competitions the winners make steady and reliable progress in, compared to other competitions which look noisier...
\n\nOr you could try forking this one and setting your own username at the top to see your teams progress over time :) Congrats on the bronze top 100 finish!
", "votes": null, "replies": [] }, { "id": 677836, "author_name": "manojprabhaakr", "author_url": "", "post_date": "11/20/2019 16:34:03", "content": "Thanks for your wishes :).
\n\nSure @jtrotman . Will do some analysis on this during this weekend. Trusting Public LB vs CV and share one more kernel. :).
", "votes": null, "replies": [] } ] }, { "id": 677451, "author_name": "datomi", "author_url": "", "post_date": "11/20/2019 07:03:50", "content": "Thanks for the amazing visualization
", "votes": null, "replies": [] }, { "id": 678156, "author_name": "boydbigdatarpg", "author_url": "", "post_date": "11/21/2019 03:18:34", "content": "Easy for understanding \nI really love Scatter Plot > <\nThank you for Visualization
", "votes": null, "replies": [] }, { "id": 678428, "author_name": "doncalculator", "author_url": "", "post_date": "11/21/2019 11:34:25", "content": "I noticed some points clustering, are those the results of the same public kernel with different hyperparameters?
\n\n
Yes - exactly that - there are always a lot of people simply resubmitting public Notebooks, hoping for a quick/easy bronze I guess, or maybe as a 'backup' submission, which their other selection fails to beat.
\n\nThe main block seems to be 130 teams with private score 0.64626, from 534 - 663. They all dropped down into those positions. Tied scores are supposed to be sorted by time so there may be some difference in public/private sorting.
\n\nWith 5dp scores there could be accidental collisions, but Meta Kaggle has full precision scores in the Submissions.csv - you can detect true copies that way. I have some code that counts numbers of 'cheap' submissions per user (public and private scores seen more than once) - some have so many they are clearly auto-scraping/polling the kernel listings and submitting the best kernels automatically (and blends of the best) - in a chase for ranking points! Feels like veering into the taking-it-all-too-seriously to publish it though ;)
\n\nSubmitting public submissions does occasionally win medals, the best (\"worst?\") example is the recent Quora code competition - there's a huge streak of over 200 identical private scores starting in bronze & ending outside it - all from one shared Notebook. (I wonder if Kaggle even ran it for each team or just duplicated the result?!)
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "676050": "Congratulations to all medalists and more generally all who got better at iterating and improving their solutions along the way :)\n\nHere's a shake-up scatter plot for *Understanding Clouds from Satellite Images* - this shows public vs private leaderboard ranks (top/left are drops and bottom/right are rises *upwards*).\n\n\n\nPerhaps some day Kaggle will build this into their UI for competitions, but until that day arrives you can find them all generated in [a kernel][1] over in the [Meta Kaggle dataset][3] that creates shake-up scatter plots for nearly all competitions. (See also [this one][2] for tables.) Check them out to see how this competition compares...\n\nThe stats are:\n - shake-up = 0.07132\n - top 10% shake-up = 0.02128\n - gold shake-up = 0.00762\n\nBiggest fall: 977 places (I won't point anyone out)\n\nBest rise was **DatNT**, gaining 714 places.\n\nEdit: Here's a top 10 leaper league table!\n\n - *+714* [1055 → 341] DatNT (@moewie94 )\n - *+688* [884 → 196] Cakey (@cakeisgood )\n - *+629* [870 → 241] Vitaly Agapitov (@datomi )\n - *+621* [844 → 223] Dzianis Sivets (@dsivets )\n - *+620* [856 → 236] CatEek (@cateek )\n - *+619* [906 → 287] Sizhe Li (@seaself )\n - *+615* [833 → 218] Brandon (@bhenrym14 )\n - *+609* [842 → 233] King (@zcy111222 )\n - *+608* [852 → 244] hics33 (@hics33 )\n - *+604* [882 → 278] Kenal Butani (@kenal99 @mesh19 @anjali100 )\n\n@ Newer Kagglers - the shake-up originator is [BreakfastPirate][4]... Check out his [discussions tab][4] for many similar shake-up reports/threads and some great \"choose wisely\" memes.\n\n [1]: https://www.kaggle.com/jtrotman/meta-kaggle-scatter-plot-competition-shake-up\n [2]: https://www.kaggle.com/jtrotman/meta-kaggle-competition-shake-up\n [3]: https://www.kaggle.com/kaggle/meta-kaggle\n [4]: https://www.kaggle.com/breakfastpirate/discussion\n [5]: https://www.youtube.com/watch?v=A0TalLrtZ24&t=0m51s", "676133": "Head and Tail is still .. its the tummy that shook !", "676196": "Thanks. I suggest adding histogram/density plot of shake-up differences.", "676621": "Yes, that would look good on a score vs score plot ... but this plot is rank vs rank, so the histogram of each axis will simply be uniform.\n\nThere is [another kernel][1] that shows the line traces of how the scores change going up the leaderboard for each comp - I'll run it when this comp enters into Meta Kaggle, but here is a preview (just private LB scores):\n\n\n\n [1]: https://www.kaggle.com/jtrotman/leaderboard-score-landscapes", "676903": "jtrotman : Thanks for the wonderful visualization. But would love to see an edit on who jumped the most as well :). It will give an idea on how trusting your CV score can improve your rank and overfitting Public LB can drop your rank.", "677005": "Thanks @manojprabhaakr - I added a mini league table to the main post...\n\nFor investigating CV versus LB performance there is a wealth of information in [Meta Kaggle][1] - all teams public/private submission scores are in there, for nearly all competitions. (Usually takes 1-2 days to add competitions so this one should be there by the weekend.)\n\nI made [this Notebook][3] to look at the progression over time of public/private scores for the winning team of each comp - it can help you see which types of competitions the winners make steady and reliable progress in, compared to other competitions which look noisier...\n\nOr you could [try forking this one][2] and setting your own username at the top to see your teams progress over time :) Congrats on the bronze top 100 finish!\n\n [1]: https://www.kaggle.com/kaggle/meta-kaggle\n [2]: https://www.kaggle.com/jtrotman/user-competition-submission-traces\n [3]: https://www.kaggle.com/jtrotman/winning-team-submission-traces", "677451": "Thanks for the amazing visualization", "677836": "Thanks for your wishes :). \n\nSure @jtrotman . Will do some analysis on this during this weekend. Trusting Public LB vs CV and share one more kernel. :).", "678156": "Easy for understanding \nI really love Scatter Plot > <\nThank you for Visualization", "678428": "I noticed some points clustering, are those the results of the same public kernel with different hyperparameters?\n\n", "678572": "Yes - exactly that - there are always a lot of people simply resubmitting public Notebooks, hoping for a quick/easy bronze I guess, or maybe as a 'backup' submission, which their other selection fails to beat.\n\nThe main block seems to be 130 teams with private score 0.64626, from 534 - 663. They all dropped down into those positions. Tied scores are supposed to be sorted by time so there may be some difference in public/private sorting.\n\nWith 5dp scores there *could* be accidental collisions, but [Meta Kaggle][2] has full precision scores in the Submissions.csv - you can detect true copies that way. I have some code that counts numbers of 'cheap' submissions per user (public and private scores seen more than once) - some have so many they are clearly auto-scraping/polling the kernel listings and submitting the best kernels automatically (and blends of the best) - in a chase for ranking points! Feels like veering into the *taking-it-all-too-seriously* to publish it though ;)\n\nSubmitting public submissions does occasionally win medals, the best (\"worst?\") example is the recent [Quora code competition][1] - there's a huge streak of over 200 identical private scores starting in bronze & ending outside it - all from one shared Notebook. (I wonder if Kaggle even ran it for each team or just duplicated the result?!) \n\n\n [1]: https://www.kaggle.com/jtrotman/meta-kaggle-scatter-plot-competition-shake-up#quora-insincere-questions-classification\n [2]: https://www.kaggle.com/kaggle/meta-kaggle" }, "source": "meta" }