![\"enter](\"https://storage.googleapis.com/kaggle-forum-message-attachments/328474/9422/TalkingData.png\")
\nNice script.So, the leaderboard ended up quite distorted, and some people are unhappy. Here is a way to show that at least you were up there in the gold/silver/bronze zones (or top 50%, or at least higher than you ended up) at some point in time…
\n\nIt turns out to be very easy to reconstruct the public leaderboard at any point in time using a few lines of Pandas. For newbies/those who haven’t noticed, all submissions that improve on the public LB are summarized here - linked from the leaderboard page - every row in that CSV creates a new public LB ordering. If you select by a time cutoff, then groupby teams and take the max submission score for each, and rank that, you end up with a snapshot of the public LB. I’ve put a kernel here to make it easy… The script is:
\n\n%matplotlib inline\nimport pandas as pd, numpy as np\n\ns = pd.read_csv('talkingdata-adtracking-fraud-detection-publicleaderboard.csv',\n parse_dates=['SubmissionDate'])\ncut = pd.to_datetime('2018-05-08')\ns = s.loc[s.SubmissionDate<=cut] # omit post competition subs\ns['doy'] = s.SubmissionDate.dt.dayofyear\n\ndoy2date = s.groupby('doy').SubmissionDate.max().dt.date.to_dict()\ndays = np.unique(s.doy.values)\n\ndef leaderboard(doy):\n return s.loc[s.doy<=doy].groupby('TeamName').Score.max().rank(ascending=False)\n\ndef leaderboard_rank(doy, team):\n return leaderboard(doy).get(team)\n\ndef chart_public_lb(team):\n ser = pd.Series({doy2date[doy]:leaderboard_rank(doy, team) for doy in days})\n p = ser.plot(figsize=(12,5))\n p.set_title(team + ' - TalkingData AdTracking Fraud Detection Challenge'\n + ' - Public LB Rank')\n p.invert_yaxis()\n p.grid(True)\n p.set_ylim(top=0)\n return p\n\nchart_public_lb('Dilbert') # put team name here\nThere might be slight inaccuracies due to team ups, but it looks correct to me, and tells the gist of the story. (Should work for any competition shorter than a year, if it's longer you can modify it to select by dates instead of day-of-year.)
\n\nI’ve plotted a bunch out of curiosity and it looks like there are a lot of interesting stories out there. Some end badly. Some have a happy ending. Many show how viciously relentless the pace of Kaggle competitions can be – stop submitting and you’ll slide quite far quite fast ;)
\n\nI’ll post only my own here – and to keep a positive spin on it I think it would be nicer if people stick to sharing their own journey, rather than using it to point out other people who mysteriously join & leap into the medals in the dying moments >:S
\n\nFor what it’s worth, I’m always impressed by people posting strong submissions early on in the competition, especially the first few days – it is more likely to be the result of your own work and not cheap click & submit or blending kernels (not that that's all bad, you can learn some things that way...) Or perhaps, having had the luck to have worked on a very similar task before. Either way I’m sure some people would like to be able to document that… especially if you use it to illustrate an explanation of what you did.
\n\nTo share here, you can fork my kernel and link to it, or put the graphic here.
\n\nSome instructions:
\n\nSo, the leaderboard ended up quite distorted, and some people are unhappy. Here is a way to show that at least you were up there in the gold/silver/bronze zones (or top 50%, or at least higher than you ended up) at some point in time…
\n\nIt turns out to be very easy to reconstruct the public leaderboard at any point in time using a few lines of Pandas. For newbies/those who haven’t noticed, all submissions that improve on the public LB are summarized here - linked from the leaderboard page - every row in that CSV creates a new public LB ordering. If you select by a time cutoff, then groupby teams and take the max submission score for each, and rank that, you end up with a snapshot of the public LB. I’ve put a kernel here to make it easy… The script is:
\n\n%matplotlib inline\nimport pandas as pd, numpy as np\n\ns = pd.read_csv('talkingdata-adtracking-fraud-detection-publicleaderboard.csv',\n parse_dates=['SubmissionDate'])\ncut = pd.to_datetime('2018-05-08')\ns = s.loc[s.SubmissionDate<=cut] # omit post competition subs\ns['doy'] = s.SubmissionDate.dt.dayofyear\n\ndoy2date = s.groupby('doy').SubmissionDate.max().dt.date.to_dict()\ndays = np.unique(s.doy.values)\n\ndef leaderboard(doy):\n return s.loc[s.doy<=doy].groupby('TeamName').Score.max().rank(ascending=False)\n\ndef leaderboard_rank(doy, team):\n return leaderboard(doy).get(team)\n\ndef chart_public_lb(team):\n ser = pd.Series({doy2date[doy]:leaderboard_rank(doy, team) for doy in days})\n p = ser.plot(figsize=(12,5))\n p.set_title(team + ' - TalkingData AdTracking Fraud Detection Challenge'\n + ' - Public LB Rank')\n p.invert_yaxis()\n p.grid(True)\n p.set_ylim(top=0)\n return p\n\nchart_public_lb('Dilbert') # put team name here\nThere might be slight inaccuracies due to team ups, but it looks correct to me, and tells the gist of the story. (Should work for any competition shorter than a year, if it's longer you can modify it to select by dates instead of day-of-year.)
\n\nI’ve plotted a bunch out of curiosity and it looks like there are a lot of interesting stories out there. Some end badly. Some have a happy ending. Many show how viciously relentless the pace of Kaggle competitions can be – stop submitting and you’ll slide quite far quite fast ;)
\n\nI’ll post only my own here – and to keep a positive spin on it I think it would be nicer if people stick to sharing their own journey, rather than using it to point out other people who mysteriously join & leap into the medals in the dying moments >:S
\n\nFor what it’s worth, I’m always impressed by people posting strong submissions early on in the competition, especially the first few days – it is more likely to be the result of your own work and not cheap click & submit or blending kernels (not that that's all bad, you can learn some things that way...) Or perhaps, having had the luck to have worked on a very similar task before. Either way I’m sure some people would like to be able to document that… especially if you use it to illustrate an explanation of what you did.
\n\nTo share here, you can fork my kernel and link to it, or put the graphic here.
\n\nSome instructions:
\n\nIt's a sad story. Middle drop due to some personal business. Final drop due to high score kernel sharing. \n\nNice script.
Hey, that's a nice script. I love little kernels like that! Here's mine:
\n\n![\"enter](\"https://i.imgur.com/AOal5Re.png\")
Looks good to me, we all know what the last day drop means... Plus you got a great haul of discussion points from that, currently 4th most popular post for the competition - congrats :)
", "rawMarkdown": "Looks good to me, we all know what the last day drop means... Plus you got a great haul of discussion points from that, currently [4th most popular post][1] for the competition - congrats :)\n\n [1]: https://www.kaggle.com/c/talkingdata-adtracking-fraud-detection/discussion/56182\n", "votes": 2 } ] }, { "id": 326995, "postDate": "2018-05-10T16:18:43.333Z", "content": "Here’s mine to start off with – fuller write-up still to come…
![\"JT](\"https://storage.googleapis.com/kaggle-forum-message-attachments/326995/9398/jt_public_lb.png\")
Edit: In short, I started off with some old categorical neural network code in the first week – managed to get top 13 then went back to the Toxic Comments competition. You can see when the better Kernels start to be shared a couple of days later, the score really starts to slide. The zig-zags are all from single lightgbm models, with poor leaderboard correlation until the last two days.
", "rawMarkdown": "Here’s mine to start off with – [fuller write-up][1]\n\n\nfuller writeup still to come…
\n
Can't wait....
", "rawMarkdown": "> fuller writeup still to come…\n\nCan't wait....", "votes": 1 }, { "id": 327051, "postDate": "2018-05-10T18:37:47.690Z", "content": "Ha, don't get too excited, I'm struggling for things to say that have not been shared already :) I have no killer tips, just odds and ends. I loved your write-up - very extensive, and I'm still smiling about the \"solo fools\" comment, congrats on your second solo gold from a (mostly silver) fellow solo fool :D
\n\n(My prediction from five months ago is looking really good!)
", "rawMarkdown": "Ha, don't get too excited, I'm struggling for things to say that have not been shared already :) I have no killer tips, just odds and ends. I loved your write-up - very extensive, and I'm **still** smiling about the \"solo fools\" comment, congrats on your second solo gold from a (mostly silver) fellow solo fool :D\n\n([My prediction][1] from five months ago is looking really good!)\n\n [1]: https://www.kaggle.com/c/santa-gift-matching/discussion/45955#259560\n", "votes": 1 }, { "id": 327061, "postDate": "2018-05-10T19:03:32.867Z", "content": "I can make the same answer as well!
\n\nYour jump from public to private is a sign of your sound approach.
\n\nIt is not the first time. This clearly shows luck does not play a role in your results.
", "rawMarkdown": "I can make the [same answer][1] as well!\n\nYour jump from public to private is a sign of your sound approach. \n\nIt is not the first time. This clearly shows luck does not play a role in your results.\n\n [1]: https://www.kaggle.com/c/santa-gift-matching/discussion/45955#259569", "votes": 1 }, { "id": 327294, "postDate": "2018-05-11T07:43:35.917Z", "content": "You also made another prediction that almost came true here!
", "rawMarkdown": "You also made [another prediction][1] that almost came true here!\n\n\n [1]: https://www.kaggle.com/c/santa-gift-matching/discussion/45955#259542", "votes": 1 }, { "id": 327443, "postDate": "2018-05-11T15:06:40.800Z", "content": "It is really interesting. I was thinking then about how fragile the whole setup can be, where one defector can ruin the concept.
\n\nI posted an idea in the middle of the megathread about implicit team merging - identical CSV files are a special case of highly correlated submissions that it would be quite easy to address, by merging them into one implicit team.
\n\nIdentical submissions are easy to detect but once you look at high correlations it becomes shades of grey, harder to make policies. (Also I can't recall seeing so few leaderboard removals from a competition as this one, my guess is, since scoring takes four minutes or so, and the system cannot take the original test set, the cheater detection system is perhaps not efficient enough to run in the time available?!)
\n\nTo some extent the sharing problem is self balancing - I can't recall the top 5 ever sharing a crucial secret, or the top 10, except perhaps in Quora, but top 100 tips are fairly regular, so those still leave a huge majority of people affected by it and very disappointed - it looks like Dirk deleted his account after the negative feedback. So it won't be him next time ;)
\n\nIt is hard for Kaggle... Competitors and sponsors have different goals - I think ideas to lock things down a bit in the last week are good, but I'm not sure a competition sponsor would like the idea of limiting progress...
", "rawMarkdown": "It is really interesting. I was thinking then about how fragile the whole setup can be, where one defector can ruin the concept.\n\nI posted an [idea in the middle of the megathread][1] about implicit team merging - identical CSV files are a special case of highly correlated submissions that it would be quite easy to address, by merging them into one implicit team.\n\nIdentical submissions are easy to detect but once you look at high correlations it becomes shades of grey, harder to make policies. (Also I can't recall seeing so few leaderboard removals from a competition as this one, my guess is, since scoring takes four minutes or so, and the [system cannot take the original test set][2], the cheater detection system is perhaps not efficient enough to run in the time available?!)\n\nTo some extent the sharing problem is self balancing - I can't recall the top 5 ever sharing a crucial secret, or the top 10, except perhaps in [Quora][3], but top 100 tips are fairly regular, so those still leave a huge majority of people affected by it and very disappointed - it looks like Dirk deleted his account after the negative feedback. So it won't be him next time ;)\n\nIt is hard for Kaggle... Competitors and sponsors have different goals - I think ideas to lock things down a bit in the last week are good, but I'm not sure a competition sponsor would like the idea of limiting progress...\n\n\n [1]: https://www.kaggle.com/c/talkingdata-adtracking-fraud-detection/discussion/56182#325640\n [2]: https://www.kaggle.com/c/talkingdata-adtracking-fraud-detection/discussion/51877#295772\n [3]: https://www.kaggle.com/c/quora-question-pairs/discussion/33801\n", "votes": 2 }, { "id": 327446, "postDate": "2018-05-11T15:09:58.900Z", "content": "I like your mega team idea: identical subs should be merged into one team. Simple and efficient. It should make people think twice about reusing a kernel output as it.
", "rawMarkdown": "I like your mega team idea: identical subs should be merged into one team. Simple and efficient. It should make people think twice about reusing a kernel output as it.", "votes": 1 }, { "id": 327447, "postDate": "2018-05-11T15:15:34.200Z", "content": "Yes, I forgot to add that obviously it just shifts the problem along a bit, people add noise, change single rows or average with other subs, but it often surprises me how large the clumps of identical scores are, and even if those clumps were merged into teams and had lower points I'd bet some people would still do it.
", "rawMarkdown": "Yes, I forgot to add that obviously it just shifts the problem along a bit, people add noise, change single rows or average with other subs, but it often surprises me how large the clumps of identical scores are, and even if those clumps were merged into teams and had lower points I'd bet some people would still do it." }, { "id": 327454, "postDate": "2018-05-11T15:32:13.167Z", "content": "Yes, but at least it impacts people outside the mega team by only one rank shift instead of a very large number. And yes, it can be defeated by modifying the submission. At least it filters out the cut and paste specialists who are a plague here.
", "rawMarkdown": "Yes, but at least it impacts people outside the mega team by only one rank shift instead of a very large number. And yes, it can be defeated by modifying the submission. At least it filters out the cut and paste specialists who are a plague here.", "votes": 1 } ] }, { "id": 328357, "postDate": "2018-05-14T04:09:40.057Z", "content": "That is interesting!\n![\"enter](\"https://storage.googleapis.com/kaggle-forum-message-attachments/328357/9420/lb.png\")
Congrats on the gold finish! Also, since the dynamic leaderboard pages only load the top 50 by default, it's a nice achievement to stay on the 'front page' of the leaderboard for so long :)
\n\nI'm curious to know more about your solution, e.g. were matrix factorizations part of it or did you have other techniques?
", "rawMarkdown": "Congrats on the gold finish! Also, since the dynamic leaderboard pages only load the top 50 by default, it's a nice achievement to stay on the 'front page' of the leaderboard for so long :)\n\nI'm curious to know more about your solution, e.g. were matrix factorizations part of it or did you have other techniques?\n" }, { "id": 328624, "postDate": "2018-05-14T18:19:42.227Z", "content": "My solution is nothing special, so I didn't post it. Around 40 simple features (most of them can be found in kernels) and carefully tuned lightgbm. Single model, no ensemble, no ordering trick.
", "rawMarkdown": "My solution is nothing special, so I didn't post it. Around 40 simple features (most of them can be found in kernels) and carefully tuned lightgbm. Single model, no ensemble, no ordering trick.", "votes": 4 } ] }, { "id": 327860, "postDate": "2018-05-12T17:40:51.857Z", "content": "![\"\"](\"https://storage.googleapis.com/kaggle-forum-message-attachments/327860/9408/__results___4_1.png\")
Here is our story
Hi Araks - You got some well earned discussion medals from this competition, I liked your From 100k to 100MM rows post early on.
\n\nThe graph looks like a nice progression to me, you mentioned elsewhere that your private leaderboard drop is partly* due to not selecting two submissions, which is a vital part of every competition... play some more and you may see how frustrating a process it can be...
\n\nThere are a lot of good threads in different competitions about this public → private shake-up, a particularly good one is the Shake-up??? thread by @BreakfastPirate in the Bosch competition.
\n\nMy favourite comment there is quoted below... (it may take you some more competitions to find this as funny as long time Kagglers will :)
\n\n\n\n\nBreakfastPirate wrote
\n \nWhen you select your 2 submissions, choose wisely. Don't choose poorly. There is a lot of Kaggle-wisdom in this video clip: https://www.youtube.com/watch?v=A0TalLrtZ24&t=0m51s
\n
I agree 100% with your comment below, I was trying to focus on positives, but you're right, for example there are 113 teams ahead of you with a score of 0.982062 from a blending kernel, which frankly sucks. They should share one rank and you should get a medal :)
\n\nBut on the positive side again, sharing insightful posts during a competition is far stronger evidence of honest hard work at trying to understand the problem and find a good model, so congrats on that :)
", "rawMarkdown": "Hi Araks - You got some well earned discussion medals from this competition, I liked your [From 100k to 100MM rows][1] post early on.\n\nThe graph looks like a nice progression to me, you mentioned elsewhere that your private leaderboard drop **is partly* due to** [not selecting two submissions][2], which is a vital part of every competition... play some more and you may see how frustrating a process it can be...\n\nThere are a lot of good threads in different competitions about this public → private shake-up, a particularly good one is the [Shake-up???][3] thread by @BreakfastPirate in the Bosch competition.\n\nMy favourite [comment there][4] is quoted below... (it may take you some more competitions to find this as funny as long time Kagglers will :)\n\n> **BreakfastPirate wrote**\n> \n> When you select your 2 submissions, choose wisely. Don't choose poorly. There is a lot of Kaggle-wisdom in this video clip: https://www.youtube.com/watch?v=A0TalLrtZ24&t=0m51s\n\n### EDIT *\n\nI agree ***100%*** with your comment below, I was trying to focus on positives, but you're right, for example there are ***113*** teams ahead of you with a score of ***0.982062*** from a blending kernel, which frankly ***sucks***. They should [share one rank][5] and you should get a medal :)\n\nBut on the positive side again, sharing insightful posts during a competition is ***far*** stronger evidence of honest hard work at trying to understand the problem and find a good model, so congrats on that :)\n\n\n [1]: https://www.kaggle.com/c/talkingdata-adtracking-fraud-detection/forums/t/52927/from-100k-to-100mm-rows\n [2]: https://www.kaggle.com/c/talkingdata-adtracking-fraud-detection/forums/t/56241/130-copied-submissions-in-silver-and-bronze?forumMessageId=326323#post326323\n [3]: https://www.kaggle.com/c/bosch-production-line-performance/discussion/24961\n [4]: https://www.kaggle.com/c/bosch-production-line-performance/discussion/24961#144008\n [5]: https://www.kaggle.com/c/talkingdata-adtracking-fraud-detection/discussion/56182#325640\n", "votes": 1 }, { "id": 327931, "postDate": "2018-05-13T00:41:13.287Z", "content": "Hi @James, thanks for looking at our progress carefully. Not selecting submissions was a huge mistake which I will try to correct next time. Another mistake would be putting too much pressure on the last days. And another mistake would be not selecting features wisely. Another mistake...
\n\nEventually, we all know the biggest reason why we had such a big drop and didn't get a medal, the last day kernels. I would be much happier to see it mentioned in your message above as well. Because if we don't acknowledge that there was such a thing, we will not form a community where everyone is ready to ignore the last day kernels even if those kernels are scoring high.
", "rawMarkdown": "Hi @James, thanks for looking at our progress carefully. Not selecting submissions was a huge mistake which I will try to correct next time. Another mistake would be putting too much pressure on the last days. And another mistake would be not selecting features wisely. Another mistake...\n \nEventually, we all know the biggest reason why we had such a big drop and didn't get a medal, the last day kernels. I would be much happier to see it mentioned in your message above as well. Because if we don't acknowledge that there was such a thing, we will not form a community where everyone is ready to ignore the last day kernels even if those kernels are scoring high.", "votes": 2 }, { "id": 329197, "postDate": "2018-05-16T00:49:06.450Z", "content": "Thanks for the edit, @James. I totally understood why you didn't mention it at first. And I see that you talk about it a lot. The thing is, I am also trying to focus on the positive but this time I am not able to forget the negative very quickly and want everyone to remember it with me, haha. I needed a medal for my resume (and only for that, honestly). So now I hope that at least the tremendous learning experience will pay off.
", "rawMarkdown": "Thanks for the edit, @James. I totally understood why you didn't mention it at first. And I see that you talk about it a lot. The thing is, I am also trying to focus on the positive but this time I am not able to forget the negative very quickly and want everyone to remember it with me, haha. I needed a medal for my resume (and only for that, honestly). So now I hope that at least the tremendous learning experience will pay off. ", "votes": 1 }, { "id": 329201, "postDate": "2018-05-16T00:52:27.620Z", "content": "By the way, I have already upvoted your comment about sharing one rank and it is a very clever solution to this situation.
", "rawMarkdown": "By the way, I have already upvoted your comment about sharing one rank and it is a very clever solution to this situation.", "votes": 1 }, { "id": 329538, "postDate": "2018-05-16T17:03:33.860Z", "content": "I hope it does pay off for you, despite the drawbacks of disruptive late sharing Kaggle is still a great site to try things out and learn.
\n\nPractice makes perfect and if you come back and try another competition you'll already be ahead of the many people who do only one competition (interesting kernel), it's only a relatively small number of us (odd) heavy users that keep coming back for more :)
\n\n(Also - you can still make submissions after the competition, so you can see which of the shared ideas work best with your features. In some way it's more worthwhile - after deadline day you're obviously not doing it for the medal, purely for the knowledge...)
", "rawMarkdown": "I hope it does pay off for you, despite the drawbacks of disruptive late sharing Kaggle is still a great site to try things out and learn.\n\n*Practice makes perfect* and if you come back and try another competition you'll already be ahead of the [many people who do only one competition][1] (interesting kernel), it's only a relatively small number of us (odd) heavy users that keep coming back for more :)\n\n(Also - you can still make submissions after the competition, so you can see which of the shared ideas work best with your features. In some way it's *more* worthwhile - after deadline day you're obviously not doing it for the medal, purely for the knowledge...)\n\n [1]: https://www.kaggle.com/mlearn/user-engagement-on-kaggle-competitions\n\n", "votes": 1 } ] }, { "id": 330089, "postDate": "2018-05-18T02:55:24.413Z", "content": "Nice script! Here is mine:
\n\n![\"ShawnXiao\"](\"https://storage.googleapis.com/kaggle-forum-message-attachments/330089/9443/ShawnXiao.png\")
Great Script!! Here is our Public LB Journey. were constantly in Top 150, before high score kernel sharing. Also better submission selection would have us within ~100. \n![\"Here](\"https://storage.googleapis.com/kaggle-forum-message-attachments/328217/9421/__results___4_1.png\")
Thanks for sharing Vikas - lots of up-ticks in the graph there, makes me think that a metric that summarizes the gradients in the line would be another indicator of performance and/or effort...
\n\nActually, I see you have made a public fork of the kernel and linked to the image output, which due to a Kaggle bug stops working after a short while. I definitely saw the graph on this page earlier but now I see the img alt-text \"Here is our Public LB Journey\"...
", "rawMarkdown": "Thanks for sharing Vikas - lots of up-ticks in the graph there, makes me think that a metric that summarizes the gradients in the line would be another indicator of performance and/or effort...\n\nActually, I see you have made a [public fork][1] of the kernel and linked to the image output, which due to a Kaggle bug stops working after a short while. I definitely saw the graph on this page earlier but now I see the img alt-text \"Here is our Public LB Journey\"...\n\n [1]: https://www.kaggle.com/vikasp/talkingdata-public-lb-progression-my-team\n", "votes": 1 }, { "id": 328362, "postDate": "2018-05-14T04:21:00.880Z", "content": "Thanks for the tip, fixed :)
", "rawMarkdown": "Thanks for the tip, fixed :)" }, { "id": 333288, "postDate": "2018-05-24T20:01:52.083Z", "content": "This script is a such a relief , i know people with 3 submissions and a bronze , waiting for them to publish the LB journey.
", "rawMarkdown": "This script is a such a relief , i know people with 3 submissions and a bronze , waiting for them to publish the LB journey.", "votes": 1 } ] }, { "id": 327495, "postDate": "2018-05-11T17:21:47.707Z", "content": "Hi my evolution , downhill
", "rawMarkdown": "Hi my evolution , downhill", "votes": 2, "replies": [ { "id": 327532, "postDate": "2018-05-11T19:42:41.973Z", "content": "Hi Bruno16 - thanks for sharing, although I can't see an image in your post, I still have the notebook open so I've just generated it myself - some nice leaps up the order there :)
\n\n![\"Bruno16](\"https://i.imgur.com/4nCdMN4.png\")
Thanks James, the upload button was not working...
", "rawMarkdown": "Thanks James, the upload button was not working..." } ] }, { "id": 327055, "postDate": "2018-05-10T18:51:13.530Z", "content": "![\"enter](\"https://storage.googleapis.com/kaggle-forum-message-attachments/327055/9402/index.png\")
very nice kernel James!
Thanks @steubk! That's a really consistent performance, and it makes me realise version 2 should have markers indicating when a team's actual submissions are made. In many other plots the time of most submissions are obvious from the up-tick in the line but from your graph I can't tell. That would make for another possible metric of performance - leaderboard rank drift over time vs submission rate...
", "rawMarkdown": "Thanks @steubk! That's a really consistent performance, and it makes me realise version 2 should have markers indicating when a team's actual submissions are made. In many other plots the time of most submissions are obvious from the up-tick in the line but from your graph I can't tell. That would make for another possible metric of performance - leaderboard rank drift over time vs submission rate...", "votes": 1 } ] }, { "id": 327019, "postDate": "2018-05-10T17:25:34.460Z", "content": "Great, thanks. Here is mine. I think the drop is when I worked on DSB competition.
\n\n![\"enter](\"https://storage.googleapis.com/kaggle-forum-message-attachments/327019/9401/cpmp.png\")
Ha, I was beating you so badly on 2018-03-23, you noob!
\n\ncries at the corner of shower
", "rawMarkdown": "Ha, I was beating you so badly on 2018-03-23, you noob!\n\n*cries at the corner of shower*", "votes": 2 } ] }, { "id": 328135, "postDate": "2018-05-13T13:01:48.457Z", "content": "Nice kernel. Here's my first Kaggle competition.\n![\"enter](\"https://storage.googleapis.com/kaggle-forum-message-attachments/328135/9415/leaderboard.png\")
Congrats on getting a silver in your first competition, you managed to out-score the blenders and Dirk's army :)
\n\nAlso, you must have used a more recent copy of talkingdata-adtracking-fraud-detection-publicleaderboard.csv (the file keeps growing as people keep submitting after the competition), so your graph extends a bit longer than the May 8th deadline. I've added a fix above and in the kernel.
", "rawMarkdown": "Congrats on getting a silver in your first competition, you managed to out-score the blenders and Dirk's army :)\n\nAlso, you must have used a more recent copy of *talkingdata-adtracking-fraud-detection-publicleaderboard.csv* (the file keeps growing as people keep submitting after the competition), so your graph extends a bit longer than the May 8th deadline. I've added a fix above and in the [kernel][1].\n\n\n [1]: https://www.kaggle.com/jtrotman/show-your-public-lb-progression-over-time", "votes": 1 }, { "id": 328218, "postDate": "2018-05-13T17:38:41.180Z", "content": "Thanks James. \nI've fixed it.\n![\"enter](\"https://storage.googleapis.com/kaggle-forum-message-attachments/328135/9418/leaderboard_5.8.png\")
It's a sad story. Middle drop due to some personal business. Final drop due to high score kernel sharing. \n\nNice script.
Hey, that's a nice script. I love little kernels like that! Here's mine:
\n\n
Looks good to me, we all know what the last day drop means... Plus you got a great haul of discussion points from that, currently 4th most popular post for the competition - congrats :)
", "votes": 2, "replies": [] } ] }, { "id": 326995, "author_name": "James Trotman", "author_url": "", "post_date": "2018-05-10T16:18:43.333000", "content": "Here’s mine to start off with – fuller write-up still to come…
Edit: In short, I started off with some old categorical neural network code in the first week – managed to get top 13 then went back to the Toxic Comments competition. You can see when the better Kernels start to be shared a couple of days later, the score really starts to slide. The zig-zags are all from single lightgbm models, with poor leaderboard correlation until the last two days.
", "votes": 4, "replies": [ { "id": 327041, "author_name": "CPMP", "author_url": "", "post_date": "2018-05-10T18:16:55.317000", "content": "\n\n\nfuller writeup still to come…
\n
Can't wait....
", "votes": 1, "replies": [] }, { "id": 327051, "author_name": "James Trotman", "author_url": "", "post_date": "2018-05-10T18:37:47.690000", "content": "Ha, don't get too excited, I'm struggling for things to say that have not been shared already :) I have no killer tips, just odds and ends. I loved your write-up - very extensive, and I'm still smiling about the \"solo fools\" comment, congrats on your second solo gold from a (mostly silver) fellow solo fool :D
\n\n(My prediction from five months ago is looking really good!)
", "votes": 1, "replies": [] }, { "id": 327061, "author_name": "CPMP", "author_url": "", "post_date": "2018-05-10T19:03:32.867000", "content": "I can make the same answer as well!
\n\nYour jump from public to private is a sign of your sound approach.
\n\nIt is not the first time. This clearly shows luck does not play a role in your results.
", "votes": 1, "replies": [] }, { "id": 327294, "author_name": "CPMP", "author_url": "", "post_date": "2018-05-11T07:43:35.917000", "content": "You also made another prediction that almost came true here!
", "votes": 1, "replies": [] }, { "id": 327443, "author_name": "James Trotman", "author_url": "", "post_date": "2018-05-11T15:06:40.800000", "content": "It is really interesting. I was thinking then about how fragile the whole setup can be, where one defector can ruin the concept.
\n\nI posted an idea in the middle of the megathread about implicit team merging - identical CSV files are a special case of highly correlated submissions that it would be quite easy to address, by merging them into one implicit team.
\n\nIdentical submissions are easy to detect but once you look at high correlations it becomes shades of grey, harder to make policies. (Also I can't recall seeing so few leaderboard removals from a competition as this one, my guess is, since scoring takes four minutes or so, and the system cannot take the original test set, the cheater detection system is perhaps not efficient enough to run in the time available?!)
\n\nTo some extent the sharing problem is self balancing - I can't recall the top 5 ever sharing a crucial secret, or the top 10, except perhaps in Quora, but top 100 tips are fairly regular, so those still leave a huge majority of people affected by it and very disappointed - it looks like Dirk deleted his account after the negative feedback. So it won't be him next time ;)
\n\nIt is hard for Kaggle... Competitors and sponsors have different goals - I think ideas to lock things down a bit in the last week are good, but I'm not sure a competition sponsor would like the idea of limiting progress...
", "votes": 2, "replies": [] }, { "id": 327446, "author_name": "CPMP", "author_url": "", "post_date": "2018-05-11T15:09:58.900000", "content": "I like your mega team idea: identical subs should be merged into one team. Simple and efficient. It should make people think twice about reusing a kernel output as it.
", "votes": 1, "replies": [] }, { "id": 327447, "author_name": "James Trotman", "author_url": "", "post_date": "2018-05-11T15:15:34.200000", "content": "Yes, I forgot to add that obviously it just shifts the problem along a bit, people add noise, change single rows or average with other subs, but it often surprises me how large the clumps of identical scores are, and even if those clumps were merged into teams and had lower points I'd bet some people would still do it.
", "votes": 0, "replies": [] }, { "id": 327454, "author_name": "CPMP", "author_url": "", "post_date": "2018-05-11T15:32:13.167000", "content": "Yes, but at least it impacts people outside the mega team by only one rank shift instead of a very large number. And yes, it can be defeated by modifying the submission. At least it filters out the cut and paste specialists who are a plague here.
", "votes": 1, "replies": [] } ] }, { "id": 328357, "author_name": "Cheng", "author_url": "", "post_date": "2018-05-14T04:09:40.057000", "content": "That is interesting!\n
Congrats on the gold finish! Also, since the dynamic leaderboard pages only load the top 50 by default, it's a nice achievement to stay on the 'front page' of the leaderboard for so long :)
\n\nI'm curious to know more about your solution, e.g. were matrix factorizations part of it or did you have other techniques?
", "votes": 0, "replies": [] }, { "id": 328624, "author_name": "Cheng", "author_url": "", "post_date": "2018-05-14T18:19:42.227000", "content": "My solution is nothing special, so I didn't post it. Around 40 simple features (most of them can be found in kernels) and carefully tuned lightgbm. Single model, no ensemble, no ordering trick.
", "votes": 4, "replies": [] } ] }, { "id": 327860, "author_name": "Araks Stepanyan", "author_url": "", "post_date": "2018-05-12T17:40:51.857000", "content": " Here is our story
Hi Araks - You got some well earned discussion medals from this competition, I liked your From 100k to 100MM rows post early on.
\n\nThe graph looks like a nice progression to me, you mentioned elsewhere that your private leaderboard drop is partly* due to not selecting two submissions, which is a vital part of every competition... play some more and you may see how frustrating a process it can be...
\n\nThere are a lot of good threads in different competitions about this public → private shake-up, a particularly good one is the Shake-up??? thread by @BreakfastPirate in the Bosch competition.
\n\nMy favourite comment there is quoted below... (it may take you some more competitions to find this as funny as long time Kagglers will :)
\n\n\n\n\nBreakfastPirate wrote
\n \nWhen you select your 2 submissions, choose wisely. Don't choose poorly. There is a lot of Kaggle-wisdom in this video clip: https://www.youtube.com/watch?v=A0TalLrtZ24&t=0m51s
\n
I agree 100% with your comment below, I was trying to focus on positives, but you're right, for example there are 113 teams ahead of you with a score of 0.982062 from a blending kernel, which frankly sucks. They should share one rank and you should get a medal :)
\n\nBut on the positive side again, sharing insightful posts during a competition is far stronger evidence of honest hard work at trying to understand the problem and find a good model, so congrats on that :)
", "votes": 1, "replies": [] }, { "id": 327931, "author_name": "Araks Stepanyan", "author_url": "", "post_date": "2018-05-13T00:41:13.287000", "content": "Hi @James, thanks for looking at our progress carefully. Not selecting submissions was a huge mistake which I will try to correct next time. Another mistake would be putting too much pressure on the last days. And another mistake would be not selecting features wisely. Another mistake...
\n\nEventually, we all know the biggest reason why we had such a big drop and didn't get a medal, the last day kernels. I would be much happier to see it mentioned in your message above as well. Because if we don't acknowledge that there was such a thing, we will not form a community where everyone is ready to ignore the last day kernels even if those kernels are scoring high.
", "votes": 2, "replies": [] }, { "id": 329197, "author_name": "Araks Stepanyan", "author_url": "", "post_date": "2018-05-16T00:49:06.450000", "content": "Thanks for the edit, @James. I totally understood why you didn't mention it at first. And I see that you talk about it a lot. The thing is, I am also trying to focus on the positive but this time I am not able to forget the negative very quickly and want everyone to remember it with me, haha. I needed a medal for my resume (and only for that, honestly). So now I hope that at least the tremendous learning experience will pay off.
", "votes": 1, "replies": [] }, { "id": 329201, "author_name": "Araks Stepanyan", "author_url": "", "post_date": "2018-05-16T00:52:27.620000", "content": "By the way, I have already upvoted your comment about sharing one rank and it is a very clever solution to this situation.
", "votes": 1, "replies": [] }, { "id": 329538, "author_name": "James Trotman", "author_url": "", "post_date": "2018-05-16T17:03:33.860000", "content": "I hope it does pay off for you, despite the drawbacks of disruptive late sharing Kaggle is still a great site to try things out and learn.
\n\nPractice makes perfect and if you come back and try another competition you'll already be ahead of the many people who do only one competition (interesting kernel), it's only a relatively small number of us (odd) heavy users that keep coming back for more :)
\n\n(Also - you can still make submissions after the competition, so you can see which of the shared ideas work best with your features. In some way it's more worthwhile - after deadline day you're obviously not doing it for the medal, purely for the knowledge...)
", "votes": 1, "replies": [] } ] }, { "id": 330089, "author_name": "Shawn Xiao", "author_url": "", "post_date": "2018-05-18T02:55:24.413000", "content": "Nice script! Here is mine:
\n\n
Great Script!! Here is our Public LB Journey. were constantly in Top 150, before high score kernel sharing. Also better submission selection would have us within ~100. \n
Thanks for sharing Vikas - lots of up-ticks in the graph there, makes me think that a metric that summarizes the gradients in the line would be another indicator of performance and/or effort...
\n\nActually, I see you have made a public fork of the kernel and linked to the image output, which due to a Kaggle bug stops working after a short while. I definitely saw the graph on this page earlier but now I see the img alt-text \"Here is our Public LB Journey\"...
", "votes": 1, "replies": [] }, { "id": 328362, "author_name": "Vikas Pandey", "author_url": "", "post_date": "2018-05-14T04:21:00.880000", "content": "Thanks for the tip, fixed :)
", "votes": 0, "replies": [] }, { "id": 333288, "author_name": "Mayank Soni", "author_url": "", "post_date": "2018-05-24T20:01:52.083000", "content": "This script is a such a relief , i know people with 3 submissions and a bronze , waiting for them to publish the LB journey.
", "votes": 1, "replies": [] } ] }, { "id": 327495, "author_name": "Bruno16", "author_url": "", "post_date": "2018-05-11T17:21:47.707000", "content": "Hi my evolution , downhill
", "votes": 2, "replies": [ { "id": 327532, "author_name": "James Trotman", "author_url": "", "post_date": "2018-05-11T19:42:41.973000", "content": "Hi Bruno16 - thanks for sharing, although I can't see an image in your post, I still have the notebook open so I've just generated it myself - some nice leaps up the order there :)
\n\n
Thanks James, the upload button was not working...
", "votes": 0, "replies": [] } ] }, { "id": 327055, "author_name": "steubk", "author_url": "", "post_date": "2018-05-10T18:51:13.530000", "content": "very nice kernel James!
Thanks @steubk! That's a really consistent performance, and it makes me realise version 2 should have markers indicating when a team's actual submissions are made. In many other plots the time of most submissions are obvious from the up-tick in the line but from your graph I can't tell. That would make for another possible metric of performance - leaderboard rank drift over time vs submission rate...
", "votes": 1, "replies": [] } ] }, { "id": 327019, "author_name": "CPMP", "author_url": "", "post_date": "2018-05-10T17:25:34.460000", "content": "Great, thanks. Here is mine. I think the drop is when I worked on DSB competition.
\n\n
Ha, I was beating you so badly on 2018-03-23, you noob!
\n\ncries at the corner of shower
", "votes": 2, "replies": [] } ] }, { "id": 328135, "author_name": "Shinan Xu", "author_url": "", "post_date": "2018-05-13T13:01:48.457000", "content": "Nice kernel. Here's my first Kaggle competition.\n
Congrats on getting a silver in your first competition, you managed to out-score the blenders and Dirk's army :)
\n\nAlso, you must have used a more recent copy of talkingdata-adtracking-fraud-detection-publicleaderboard.csv (the file keeps growing as people keep submitting after the competition), so your graph extends a bit longer than the May 8th deadline. I've added a fix above and in the kernel.
", "votes": 1, "replies": [] }, { "id": 328218, "author_name": "Shinan Xu", "author_url": "", "post_date": "2018-05-13T17:38:41.180000", "content": "Thanks James. \nI've fixed it.\n