{ "id": 279823, "title": "The real winner ...", "url": "/competitions/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/279823", "author_name": "", "post_date": "2021-10-19T07:34:15.698869200Z", "votes": 100, "comment_count": 22, "views": 0, "content": "

With a LB score of 0.36919 (rank 1555/1555), @tommzzhou actually had the best model !

\n

His model has the same discriminative power as a 0.63081 one, which ranks #1.
\n(because auc(y, x) = 1 - auc(y, 1 - x))

\n

Which is actually quite funny !

", "messages": [ { "id": "1549761", "postDate": "10/19/2021 07:34:15", "content": "

With a LB score of 0.36919 (rank 1555/1555), @tommzzhou actually had the best model !

\n

His model has the same discriminative power as a 0.63081 one, which ranks #1.
\n(because auc(y, x) = 1 - auc(y, 1 - x))

\n

Which is actually quite funny !

", "rawMarkdown": "With a LB score of 0.36919 (rank 1555/1555), @tommzzhou actually had the best model !\n\nHis model has the same discriminative power as a 0.63081 one, which ranks #1.\n(because `auc(y, x) = 1 - auc(y, 1 - x)`)\n\nWhich is actually quite funny !", "votes": null }, { "id": "1549812", "postDate": "10/19/2021 08:04:47", "content": "

If only he had reversed all his predictions…

\n

Does that mean every single team's prediction is no better than random prediction then? And the 1st place guy is actually just the luckiest guy?

", "rawMarkdown": "If only he had reversed all his predictions...\n\nDoes that mean every single team's prediction is no better than random prediction then? And the 1st place guy is actually just the luckiest guy?", "votes": null }, { "id": "1549820", "postDate": "10/19/2021 08:10:17", "content": "

Unless he intended to score this low, the answer to both of your questions is most likely yes, unfortunately.
\nI'm still waiting for some top 10 write-ups though, there could be a bit more than luck

", "rawMarkdown": "Unless he intended to score this low, the answer to both of your questions is most likely yes, unfortunately. \nI'm still waiting for some top 10 write-ups though, there could be a bit more than luck", "votes": null }, { "id": "1549843", "postDate": "10/19/2021 08:22:16", "content": "

Sorry to disappoint you, but I've checked the 3rd place notebook and I haven't found anything particularly interesting. I doubt other solutions in top 10 would have any innovative idea.

", "rawMarkdown": "Sorry to disappoint you, but I've checked the [3rd place notebook](https://www.kaggle.com/cedricsoares/tf-efficientnet-transfer-learning-strat-split) and I haven't found anything particularly interesting. I doubt other solutions in top 10 would have any innovative idea.", "votes": null }, { "id": "1549879", "postDate": "10/19/2021 08:58:26", "content": "

I really want to know what is the secret behind 0.369 ROC AUC score. How is that possible?

", "rawMarkdown": "I really want to know what is the secret behind 0.369 ROC AUC score. How is that possible?", "votes": null }, { "id": "1549991", "postDate": "10/19/2021 10:48:20", "content": "

There is a >0.01% chance to score >0.64 with random preds so it makes sense someone got this lucky but on the other side of the gaussian

", "rawMarkdown": "There is a >0.01% chance to score >0.64 with random preds so it makes sense someone got this lucky but on the other side of the gaussian", "votes": null }, { "id": "1550015", "postDate": "10/19/2021 11:12:03", "content": "

Kaggle has (or should I say had, see below) a blanket policy for AUC competitions: invert scores that are <0.25 (to counter score hiding).

\n

(Btw, I published a notebook here showing the policy is applied separately to the public/private scores!)

\n

If the invert threshold was 0.3, @tommzzhou would be the winner now… So you could say the winner has been decided by this threshold - it's a subjective/algorithmic policy decision rather than the purely objective best score wins…

", "rawMarkdown": "Kaggle has (or should I say *had*, see below) a blanket policy for AUC competitions: invert scores that are <0.25 (to counter score hiding).\n\n(Btw, I published a [notebook here](https://www.kaggle.com/jtrotman/auc-scoring-quirk) showing the policy is applied **separately** to the public/private scores!)\n\nIf the *invert* threshold was 0.3, @tommzzhou would be the winner now... So you could say the winner has been decided by this threshold - it's a subjective/algorithmic policy decision rather than the purely objective *best score wins*...", "votes": null }, { "id": "1550021", "postDate": "10/19/2021 11:20:08", "content": "

I didn't know Kaggle does that under the hood. What are they achieving with this implementation?

", "rawMarkdown": "I didn't know Kaggle does that under the hood. What are they achieving with this implementation?", "votes": null }, { "id": "1550054", "postDate": "10/19/2021 11:59:03", "content": "

Presumably it is because without this policy you could invert your predictions and get perfect public LB feedback, but without raising your LB position by posting high AUC scores… (as tried here for example ;-P) Or the story by Giulio on Quora: wait until the final week before switching to your real submissions and take everyone by surprise!

", "rawMarkdown": "Presumably it is because without this policy you could invert your predictions and get perfect public LB feedback, but without raising your LB position by posting high AUC scores... (as tried [here][1] for example ;-P) Or the story by Giulio on [Quora][2]: wait until the final week before switching to your real submissions and take everyone by surprise!\n\n[1]: https://www.kaggle.com/c/santander-customer-satisfaction/discussion/19323\n[2]: https://www.quora.com/How-did-you-become-a-Kaggle-Master-and-what-are-the-steps-resources-you-used-to-get-there", "votes": null }, { "id": "1550075", "postDate": "10/19/2021 12:25:45", "content": "

I am new to this kaggle field and I don't know what just happened. My notebook was much good and was having 0.91 as validation AUC so I thought that I would get the medal for sure but something strange happened and i got 1526 in the rank. Grandmasters, Masters can you pls elaborate it

", "rawMarkdown": "I am new to this kaggle field and I don't know what just happened. My notebook was much good and was having 0.91 as validation AUC so I thought that I would get the medal for sure but something strange happened and i got 1526 in the rank. Grandmasters, Masters can you pls elaborate it", "votes": null }, { "id": "1550125", "postDate": "10/19/2021 13:10:38", "content": "

This is not (always) true, Kaggle does not (always) invert the scores. I heard this myth before, but from when I tried it on old competitions it was not true. Or maybe it depends on the competition.

", "rawMarkdown": "This is not (always) true, Kaggle does not (always) invert the scores. I heard this myth before, but from when I tried it on old competitions it was not true. Or maybe it depends on the competition.", "votes": null }, { "id": "1550141", "postDate": "10/19/2021 13:24:13", "content": "

Same thing. Does anybody know why?

", "rawMarkdown": "Same thing. Does anybody know why?", "votes": null }, { "id": "1550156", "postDate": "10/19/2021 13:35:42", "content": "

It is not a myth! Check the notebook to see which competitions had submissions with inverted scores…

\n

Following that, Inversion acknowledged they invert scores:

\n
\n

Inversion wrote:
\n The 1 - AUC correction served a purpose at one time, but there are downsides. I'll probably remove it in the near future.

\n
\n

I'm guessing by your comment he did remove it… But regardless: such a policy could change the winner here.

\n
\n

Edit: Just checked - post-2019 competitions with AUC do have some low LB scores (under 0.25), so it seems the AUC inversion is disabled. In that sense, it has now become a myth: if pre-2019 is the mythic/prehistoric Kaggle era ;)

", "rawMarkdown": "It is not a myth! Check the [notebook](https://www.kaggle.com/jtrotman/auc-scoring-quirk) to see which competitions had submissions with inverted scores...\n\nFollowing that, [Inversion acknowledged they invert scores](https://www.kaggle.com/c/ieee-fraud-detection/discussion/111758#644502):\n\n> ***Inversion wrote:***\n> The 1 - AUC correction served a purpose at one time, but there are downsides. I'll probably remove it in the near future.\n\nI'm guessing by your comment he did remove it... But regardless: such a policy could change the winner here.\n\n___\n*Edit: Just checked - post-2019 competitions with AUC do have some low LB scores (under 0.25), so it seems the AUC inversion is disabled. In that sense, it has now become a myth: if pre-2019 is the mythic/prehistoric Kaggle era ;)*", "votes": null }, { "id": "1550333", "postDate": "10/19/2021 15:59:42", "content": "

not every time you see how the participants of the competition, after its completion, wonder by what rules this competition was held and evaluated. quite interesting, sorry.

", "rawMarkdown": "not every time you see how the participants of the competition, after its completion, wonder by what rules this competition was held and evaluated. quite interesting, sorry.", "votes": null }, { "id": "1550796", "postDate": "10/20/2021 03:05:35", "content": "

bad competition , bad organizers.

", "rawMarkdown": "bad competition , bad organizers.", "votes": null }, { "id": "1550850", "postDate": "10/20/2021 05:04:54", "content": "

Wait so like our AOC now is 0.46, so does that mean earlier, b4 inversion it was 0.04?

", "rawMarkdown": "Wait so like our AOC now is 0.46, so does that mean earlier, b4 inversion it was 0.04?", "votes": null }, { "id": "1551124", "postDate": "10/20/2021 10:27:46", "content": "

@keagle please see my edit above - it seems low AUC scores are no longer inverted - if I'd noticed that I wouldn't have commented! (I now realise: I have not submitted to an AUC based comp since 2019.)

\n

But to answer your question, an AUC plot image search should help explain: the area of the whole plot is 1, so an AUC of 0.46 would become (1-0.46) 0.54 after reversing the order of the predictions.

", "rawMarkdown": "keagle please see my edit above - it seems low AUC scores are no longer inverted - if I'd noticed that I wouldn't have commented! (I now realise: I have not submitted to an AUC based comp since 2019.)\n\nBut to answer your question, an [AUC plot image search](https://www.google.com/search?q=auc+plot&tbm=isch) should help explain: the area of the whole plot is 1, so an AUC of 0.46 would become (1-0.46) 0.54 after reversing the order of the predictions.", "votes": null }, { "id": "1551450", "postDate": "10/20/2021 16:20:12", "content": "

Oh ok, thanks a lot @jtrotman

\n

BTW I just realized I did a typing mistake calling 0.54, 0.04 sorry for that.

", "rawMarkdown": "Oh ok, thanks a lot @jtrotman \n\nBTW I just realized I did a typing mistake calling 0.54, 0.04 sorry for that.", "votes": null }, { "id": "1551457", "postDate": "10/20/2021 16:27:06", "content": "

great insights

", "rawMarkdown": "great insights", "votes": null }, { "id": "1551611", "postDate": "10/20/2021 18:35:36", "content": "

help needed. have just joined

", "rawMarkdown": "help needed. have just joined", "votes": null }, { "id": "1552142", "postDate": "10/21/2021 07:25:07", "content": "

It is indeed super funny :) Great catch!

", "rawMarkdown": "It is indeed super funny :) Great catch!", "votes": null }, { "id": "1555156", "postDate": "10/23/2021 16:16:36", "content": "

It could be because the evaluation dataset isn’t part of the train and test sets, so they’re checking how well your model generalizes to novel examples. Otherwise we could just return the training and test answers and get 100%.

", "rawMarkdown": "It could be because the evaluation dataset isn’t part of the train and test sets, so they’re checking how well your model generalizes to novel examples. Otherwise we could just return the training and test answers and get 100%.", "votes": null }, { "id": "1555665", "postDate": "10/24/2021 04:50:25", "content": "

great learning!!!.

", "rawMarkdown": "great learning!!!.", "votes": null } ], "comments": [ { "id": 1549812, "author_name": "nanguyen", "author_url": "", "post_date": "10/19/2021 08:04:47", "content": "

If only he had reversed all his predictions…

\n

Does that mean every single team's prediction is no better than random prediction then? And the 1st place guy is actually just the luckiest guy?

", "votes": null, "replies": [ { "id": 1549820, "author_name": "theoviel", "author_url": "", "post_date": "10/19/2021 08:10:17", "content": "

Unless he intended to score this low, the answer to both of your questions is most likely yes, unfortunately.
\nI'm still waiting for some top 10 write-ups though, there could be a bit more than luck

", "votes": null, "replies": [] }, { "id": 1549843, "author_name": "nanguyen", "author_url": "", "post_date": "10/19/2021 08:22:16", "content": "

Sorry to disappoint you, but I've checked the 3rd place notebook and I haven't found anything particularly interesting. I doubt other solutions in top 10 would have any innovative idea.

", "votes": null, "replies": [] } ] }, { "id": 1549879, "author_name": "gunesevitan", "author_url": "", "post_date": "10/19/2021 08:58:26", "content": "

I really want to know what is the secret behind 0.369 ROC AUC score. How is that possible?

", "votes": null, "replies": [ { "id": 1549991, "author_name": "theoviel", "author_url": "", "post_date": "10/19/2021 10:48:20", "content": "

There is a >0.01% chance to score >0.64 with random preds so it makes sense someone got this lucky but on the other side of the gaussian

", "votes": null, "replies": [] } ] }, { "id": 1550015, "author_name": "jtrotman", "author_url": "", "post_date": "10/19/2021 11:12:03", "content": "

Kaggle has (or should I say had, see below) a blanket policy for AUC competitions: invert scores that are <0.25 (to counter score hiding).

\n

(Btw, I published a notebook here showing the policy is applied separately to the public/private scores!)

\n

If the invert threshold was 0.3, @tommzzhou would be the winner now… So you could say the winner has been decided by this threshold - it's a subjective/algorithmic policy decision rather than the purely objective best score wins…

", "votes": null, "replies": [ { "id": 1550021, "author_name": "gunesevitan", "author_url": "", "post_date": "10/19/2021 11:20:08", "content": "

I didn't know Kaggle does that under the hood. What are they achieving with this implementation?

", "votes": null, "replies": [] }, { "id": 1550054, "author_name": "jtrotman", "author_url": "", "post_date": "10/19/2021 11:59:03", "content": "

Presumably it is because without this policy you could invert your predictions and get perfect public LB feedback, but without raising your LB position by posting high AUC scores… (as tried here for example ;-P) Or the story by Giulio on Quora: wait until the final week before switching to your real submissions and take everyone by surprise!

", "votes": null, "replies": [] }, { "id": 1550125, "author_name": "philippsinger", "author_url": "", "post_date": "10/19/2021 13:10:38", "content": "

This is not (always) true, Kaggle does not (always) invert the scores. I heard this myth before, but from when I tried it on old competitions it was not true. Or maybe it depends on the competition.

", "votes": null, "replies": [] }, { "id": 1550156, "author_name": "jtrotman", "author_url": "", "post_date": "10/19/2021 13:35:42", "content": "

It is not a myth! Check the notebook to see which competitions had submissions with inverted scores…

\n

Following that, Inversion acknowledged they invert scores:

\n
\n

Inversion wrote:
\n The 1 - AUC correction served a purpose at one time, but there are downsides. I'll probably remove it in the near future.

\n
\n

I'm guessing by your comment he did remove it… But regardless: such a policy could change the winner here.

\n
\n

Edit: Just checked - post-2019 competitions with AUC do have some low LB scores (under 0.25), so it seems the AUC inversion is disabled. In that sense, it has now become a myth: if pre-2019 is the mythic/prehistoric Kaggle era ;)

", "votes": null, "replies": [] }, { "id": 1550333, "author_name": "zaakciiru", "author_url": "", "post_date": "10/19/2021 15:59:42", "content": "

not every time you see how the participants of the competition, after its completion, wonder by what rules this competition was held and evaluated. quite interesting, sorry.

", "votes": null, "replies": [] }, { "id": 1550850, "author_name": "keagle", "author_url": "", "post_date": "10/20/2021 05:04:54", "content": "

Wait so like our AOC now is 0.46, so does that mean earlier, b4 inversion it was 0.04?

", "votes": null, "replies": [] }, { "id": 1551124, "author_name": "jtrotman", "author_url": "", "post_date": "10/20/2021 10:27:46", "content": "

@keagle please see my edit above - it seems low AUC scores are no longer inverted - if I'd noticed that I wouldn't have commented! (I now realise: I have not submitted to an AUC based comp since 2019.)

\n

But to answer your question, an AUC plot image search should help explain: the area of the whole plot is 1, so an AUC of 0.46 would become (1-0.46) 0.54 after reversing the order of the predictions.

", "votes": null, "replies": [] }, { "id": 1551450, "author_name": "keagle", "author_url": "", "post_date": "10/20/2021 16:20:12", "content": "

Oh ok, thanks a lot @jtrotman

\n

BTW I just realized I did a typing mistake calling 0.54, 0.04 sorry for that.

", "votes": null, "replies": [] } ] }, { "id": 1550075, "author_name": "adityasharma01", "author_url": "", "post_date": "10/19/2021 12:25:45", "content": "

I am new to this kaggle field and I don't know what just happened. My notebook was much good and was having 0.91 as validation AUC so I thought that I would get the medal for sure but something strange happened and i got 1526 in the rank. Grandmasters, Masters can you pls elaborate it

", "votes": null, "replies": [ { "id": 1550141, "author_name": "lililycai", "author_url": "", "post_date": "10/19/2021 13:24:13", "content": "

Same thing. Does anybody know why?

", "votes": null, "replies": [] }, { "id": 1555156, "author_name": "evancofsky", "author_url": "", "post_date": "10/23/2021 16:16:36", "content": "

It could be because the evaluation dataset isn’t part of the train and test sets, so they’re checking how well your model generalizes to novel examples. Otherwise we could just return the training and test answers and get 100%.

", "votes": null, "replies": [] } ] }, { "id": 1550796, "author_name": "dragonzhang", "author_url": "", "post_date": "10/20/2021 03:05:35", "content": "

bad competition , bad organizers.

", "votes": null, "replies": [] }, { "id": 1551457, "author_name": "skmv2022", "author_url": "", "post_date": "10/20/2021 16:27:06", "content": "

great insights

", "votes": null, "replies": [] }, { "id": 1551611, "author_name": "suhash88", "author_url": "", "post_date": "10/20/2021 18:35:36", "content": "

help needed. have just joined

", "votes": null, "replies": [] }, { "id": 1552142, "author_name": "narsil", "author_url": "", "post_date": "10/21/2021 07:25:07", "content": "

It is indeed super funny :) Great catch!

", "votes": null, "replies": [] }, { "id": 1555665, "author_name": "vaibhavbhatt111", "author_url": "", "post_date": "10/24/2021 04:50:25", "content": "

great learning!!!.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1549761": "With a LB score of 0.36919 (rank 1555/1555), @tommzzhou actually had the best model !\n\nHis model has the same discriminative power as a 0.63081 one, which ranks #1.\n(because `auc(y, x) = 1 - auc(y, 1 - x)`)\n\nWhich is actually quite funny !", "1549812": "If only he had reversed all his predictions...\n\nDoes that mean every single team's prediction is no better than random prediction then? And the 1st place guy is actually just the luckiest guy?", "1549820": "Unless he intended to score this low, the answer to both of your questions is most likely yes, unfortunately. \nI'm still waiting for some top 10 write-ups though, there could be a bit more than luck", "1549843": "Sorry to disappoint you, but I've checked the [3rd place notebook](https://www.kaggle.com/cedricsoares/tf-efficientnet-transfer-learning-strat-split) and I haven't found anything particularly interesting. I doubt other solutions in top 10 would have any innovative idea.", "1549879": "I really want to know what is the secret behind 0.369 ROC AUC score. How is that possible?", "1549991": "There is a >0.01% chance to score >0.64 with random preds so it makes sense someone got this lucky but on the other side of the gaussian", "1550015": "Kaggle has (or should I say *had*, see below) a blanket policy for AUC competitions: invert scores that are <0.25 (to counter score hiding).\n\n(Btw, I published a [notebook here](https://www.kaggle.com/jtrotman/auc-scoring-quirk) showing the policy is applied **separately** to the public/private scores!)\n\nIf the *invert* threshold was 0.3, @tommzzhou would be the winner now... So you could say the winner has been decided by this threshold - it's a subjective/algorithmic policy decision rather than the purely objective *best score wins*...", "1550021": "I didn't know Kaggle does that under the hood. What are they achieving with this implementation?", "1550054": "Presumably it is because without this policy you could invert your predictions and get perfect public LB feedback, but without raising your LB position by posting high AUC scores... (as tried [here][1] for example ;-P) Or the story by Giulio on [Quora][2]: wait until the final week before switching to your real submissions and take everyone by surprise!\n\n[1]: https://www.kaggle.com/c/santander-customer-satisfaction/discussion/19323\n[2]: https://www.quora.com/How-did-you-become-a-Kaggle-Master-and-what-are-the-steps-resources-you-used-to-get-there", "1550075": "I am new to this kaggle field and I don't know what just happened. My notebook was much good and was having 0.91 as validation AUC so I thought that I would get the medal for sure but something strange happened and i got 1526 in the rank. Grandmasters, Masters can you pls elaborate it", "1550125": "This is not (always) true, Kaggle does not (always) invert the scores. I heard this myth before, but from when I tried it on old competitions it was not true. Or maybe it depends on the competition.", "1550141": "Same thing. Does anybody know why?", "1550156": "It is not a myth! Check the [notebook](https://www.kaggle.com/jtrotman/auc-scoring-quirk) to see which competitions had submissions with inverted scores...\n\nFollowing that, [Inversion acknowledged they invert scores](https://www.kaggle.com/c/ieee-fraud-detection/discussion/111758#644502):\n\n> ***Inversion wrote:***\n> The 1 - AUC correction served a purpose at one time, but there are downsides. I'll probably remove it in the near future.\n\nI'm guessing by your comment he did remove it... But regardless: such a policy could change the winner here.\n\n___\n*Edit: Just checked - post-2019 competitions with AUC do have some low LB scores (under 0.25), so it seems the AUC inversion is disabled. In that sense, it has now become a myth: if pre-2019 is the mythic/prehistoric Kaggle era ;)*", "1550333": "not every time you see how the participants of the competition, after its completion, wonder by what rules this competition was held and evaluated. quite interesting, sorry.", "1550796": "bad competition , bad organizers.", "1550850": "Wait so like our AOC now is 0.46, so does that mean earlier, b4 inversion it was 0.04?", "1551124": "keagle please see my edit above - it seems low AUC scores are no longer inverted - if I'd noticed that I wouldn't have commented! (I now realise: I have not submitted to an AUC based comp since 2019.)\n\nBut to answer your question, an [AUC plot image search](https://www.google.com/search?q=auc+plot&tbm=isch) should help explain: the area of the whole plot is 1, so an AUC of 0.46 would become (1-0.46) 0.54 after reversing the order of the predictions.", "1551450": "Oh ok, thanks a lot @jtrotman \n\nBTW I just realized I did a typing mistake calling 0.54, 0.04 sorry for that.", "1551457": "great insights", "1551611": "help needed. have just joined", "1552142": "It is indeed super funny :) Great catch!", "1555156": "It could be because the evaluation dataset isn’t part of the train and test sets, so they’re checking how well your model generalizes to novel examples. Otherwise we could just return the training and test answers and get 100%.", "1555665": "great learning!!!." }, "source": "meta" }