Dear all,\nI am sorry for the confusion but after a discussion with Kaggle administrators and organizers, we have decided to change the evaluation metric.\nIndeed, has some of you already discussed, the evaluation was \"too lenient\" with very high scores while small decreasing pace of the loss function.\nThere are several reasons that could explain high scores, the fact that random guesser would be already at 0.85 and the fact that ROC curves usually starts almost vertically. All in all, it makes all results look very similar and give a taste that the competition was already over.
\n\nFor this reason, we have decided that, because the competition has been started since less than 2 days, it would pays off in a long terms to change the evaluation.
\n\nI apologize personally for the confusion since I bear the responsibliity for this choice that lead too a lack of clarity in the rankings.
\n\nI am sure you will keep having fun with this competition and look forward to reading discussion and proposals
", "messages": [ { "id": 826675, "postDate": "2020-04-29T19:20:05.833Z", "content": "Dear all,\nI am sorry for the confusion but after a discussion with Kaggle administrators and organizers, we have decided to change the evaluation metric.\nIndeed, has some of you already discussed, the evaluation was \"too lenient\" with very high scores while small decreasing pace of the loss function.\nThere are several reasons that could explain high scores, the fact that random guesser would be already at 0.85 and the fact that ROC curves usually starts almost vertically. All in all, it makes all results look very similar and give a taste that the competition was already over.
\n\nFor this reason, we have decided that, because the competition has been started since less than 2 days, it would pays off in a long terms to change the evaluation.
\n\nI apologize personally for the confusion since I bear the responsibliity for this choice that lead too a lack of clarity in the rankings.
\n\nI am sure you will keep having fun with this competition and look forward to reading discussion and proposals
", "rawMarkdown": "Dear all,\nI am sorry for the confusion but after a discussion with Kaggle administrators and organizers, we have decided to change the evaluation metric.\nIndeed, has some of you already discussed, the evaluation was \"too lenient\" with very high scores while small decreasing pace of the loss function.\nThere are several reasons that could explain high scores, the fact that random guesser would be already at 0.85 and the fact that ROC curves usually starts almost vertically. All in all, it makes all results look very similar and give a taste that the competition was already over.\n\nFor this reason, we have decided that, because the competition has been started since less than 2 days, it would pays off in a long terms to change the evaluation.\n\nI apologize personally for the confusion since I bear the responsibliity for this choice that lead too a lack of clarity in the rankings.\n\nI am sure you will keep having fun with this competition and look forward to reading discussion and proposals", "votes": 34 }, { "id": 826750, "postDate": "2020-04-29T20:43:35.817Z", "content": "Thanks, I think this was the right move!
\n\nThese things happen all the time in competitions and I don't think it's caused anyone any issues. But fast action was the right thing to do, so kodus. Only thing I would say is maybe pin this thread?
\n\n@kaggle team, any chance you can clear the scores of public kernels made before this change? Otherwise the kernel page will confusingly be littered with 0.99 kernels for the rest of the competiton
", "rawMarkdown": "Thanks, I think this was the right move!\n\nThese things happen all the time in competitions and I don't think it's caused anyone any issues. But fast action was the right thing to do, so kodus. Only thing I would say is maybe pin this thread?\n\n@kaggle team, any chance you can clear the scores of public kernels made before this change? Otherwise the kernel page will confusingly be littered with 0.99 kernels for the rest of the competiton", "votes": 7 }, { "id": 859325, "postDate": "2020-05-24T10:28:39.113Z", "content": "@remicogranne, you probably have some contact with kaggle team. Could you please ask them to \"clear the scores of public kernels made before this change? Otherwise the kernel page will confusingly be littered with 0.99 kernels for the rest of the competition\", as @anokas asked in his comment (and was ignored)
", "rawMarkdown": "@remicogranne, you probably have some contact with kaggle team. Could you please ask them to \"clear the scores of public kernels made before this change? Otherwise the kernel page will confusingly be littered with 0.99 kernels for the rest of the competition\", as @anokas asked in his comment (and was ignored)", "votes": 1 }, { "id": 826747, "postDate": "2020-04-29T20:38:05.850Z", "content": "Could you please tell us the current evaluation metric? So we can fix it at our local machines:)
", "rawMarkdown": "Could you please tell us the current evaluation metric? So we can fix it at our local machines:)", "votes": 1, "replies": [ { "id": 826789, "postDate": "2020-04-29T21:31:56.093Z", "content": "https://www.kaggle.com/c/alaska2-image-steganalysis/overview/evaluation its updated
", "rawMarkdown": "https://www.kaggle.com/c/alaska2-image-steganalysis/overview/evaluation its updated", "votes": 2 } ] }, { "id": 826781, "postDate": "2020-04-29T21:22:45.177Z", "content": "Agreed. Good to clear this up quickly. I was getting worried the competition was over already... I'm glad it's not!
", "rawMarkdown": "Agreed. Good to clear this up quickly. I was getting worried the competition was over already... I'm glad it's not!", "votes": 2 }, { "id": 830674, "postDate": "2020-05-02T19:16:35.370Z", "content": "Things always happen. Thank you for the fast response.
", "rawMarkdown": "Things always happen. Thank you for the fast response." }, { "id": 829844, "postDate": "2020-05-02T05:44:26.250Z", "rawMarkdown": "", "votes": -11, "isDeleted": true }, { "id": 826739, "postDate": "2020-04-29T20:33:06.993Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 826933, "postDate": "2020-04-30T00:36:11.043Z", "content": "The evaluation webpage has been updated:\nhttps://www.kaggle.com/c/alaska2-image-steganalysis/overview/evaluation\nWe have badly simplified the weights such that it almost a mere ROC with a small emphasize on the low false-positive
", "rawMarkdown": "The evaluation webpage has been updated:\n[https://www.kaggle.com/c/alaska2-image-steganalysis/overview/evaluation](https://www.kaggle.com/c/alaska2-image-steganalysis/overview/evaluation)\nWe have badly simplified the weights such that it almost a mere ROC with a small emphasize on the low false-positive", "votes": 3 } ] }, { "id": 849607, "postDate": "2020-05-15T23:22:52.343Z", "content": "Thanks for messaging.
", "rawMarkdown": "Thanks for messaging.\n" }, { "id": 827330, "postDate": "2020-04-30T08:20:20.787Z", "content": "Thanks for the fast change!
", "rawMarkdown": "Thanks for the fast change!" } ], "comments": [ { "id": 826750, "author_name": "anokas", "author_url": "", "post_date": "2020-04-29T20:43:35.817000", "content": "Thanks, I think this was the right move!
\n\nThese things happen all the time in competitions and I don't think it's caused anyone any issues. But fast action was the right thing to do, so kodus. Only thing I would say is maybe pin this thread?
\n\n@kaggle team, any chance you can clear the scores of public kernels made before this change? Otherwise the kernel page will confusingly be littered with 0.99 kernels for the rest of the competiton
", "votes": 7, "replies": [] }, { "id": 859325, "author_name": "Michael Diskin", "author_url": "", "post_date": "2020-05-24T10:28:39.113000", "content": "@remicogranne, you probably have some contact with kaggle team. Could you please ask them to \"clear the scores of public kernels made before this change? Otherwise the kernel page will confusingly be littered with 0.99 kernels for the rest of the competition\", as @anokas asked in his comment (and was ignored)
", "votes": 1, "replies": [] }, { "id": 826747, "author_name": "[ods.ai]Uladzimir Tumanau", "author_url": "", "post_date": "2020-04-29T20:38:05.850000", "content": "Could you please tell us the current evaluation metric? So we can fix it at our local machines:)
", "votes": 1, "replies": [ { "id": 826789, "author_name": "DrHB", "author_url": "", "post_date": "2020-04-29T21:31:56.093000", "content": "https://www.kaggle.com/c/alaska2-image-steganalysis/overview/evaluation its updated
", "votes": 2, "replies": [] } ] }, { "id": 826781, "author_name": "Andy Penrose", "author_url": "", "post_date": "2020-04-29T21:22:45.177000", "content": "Agreed. Good to clear this up quickly. I was getting worried the competition was over already... I'm glad it's not!
", "votes": 2, "replies": [] }, { "id": 830674, "author_name": "Shane", "author_url": "", "post_date": "2020-05-02T19:16:35.370000", "content": "Things always happen. Thank you for the fast response.
", "votes": 0, "replies": [] }, { "id": 829844, "author_name": "", "author_url": "", "post_date": "2020-05-02T05:44:26.250000", "content": "", "votes": -11, "replies": [] }, { "id": 826739, "author_name": "", "author_url": "", "post_date": "2020-04-29T20:33:06.993000", "content": "", "votes": 0, "replies": [ { "id": 826933, "author_name": "Rémi Cogranne", "author_url": "", "post_date": "2020-04-30T00:36:11.043000", "content": "The evaluation webpage has been updated:\nhttps://www.kaggle.com/c/alaska2-image-steganalysis/overview/evaluation\nWe have badly simplified the weights such that it almost a mere ROC with a small emphasize on the low false-positive
", "votes": 3, "replies": [] } ] }, { "id": 849607, "author_name": "mazalgarab", "author_url": "", "post_date": "2020-05-15T23:22:52.343000", "content": "Thanks for messaging.
", "votes": 0, "replies": [] }, { "id": 827330, "author_name": "Max Jeblick", "author_url": "", "post_date": "2020-04-30T08:20:20.787000", "content": "Thanks for the fast change!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "826675": "Dear all,\nI am sorry for the confusion but after a discussion with Kaggle administrators and organizers, we have decided to change the evaluation metric.\nIndeed, has some of you already discussed, the evaluation was \"too lenient\" with very high scores while small decreasing pace of the loss function.\nThere are several reasons that could explain high scores, the fact that random guesser would be already at 0.85 and the fact that ROC curves usually starts almost vertically. All in all, it makes all results look very similar and give a taste that the competition was already over.\n\nFor this reason, we have decided that, because the competition has been started since less than 2 days, it would pays off in a long terms to change the evaluation.\n\nI apologize personally for the confusion since I bear the responsibliity for this choice that lead too a lack of clarity in the rankings.\n\nI am sure you will keep having fun with this competition and look forward to reading discussion and proposals", "826750": "Thanks, I think this was the right move!\n\nThese things happen all the time in competitions and I don't think it's caused anyone any issues. But fast action was the right thing to do, so kodus. Only thing I would say is maybe pin this thread?\n\n@kaggle team, any chance you can clear the scores of public kernels made before this change? Otherwise the kernel page will confusingly be littered with 0.99 kernels for the rest of the competiton", "859325": "@remicogranne, you probably have some contact with kaggle team. Could you please ask them to \"clear the scores of public kernels made before this change? Otherwise the kernel page will confusingly be littered with 0.99 kernels for the rest of the competition\", as @anokas asked in his comment (and was ignored)", "826747": "Could you please tell us the current evaluation metric? So we can fix it at our local machines:)", "826781": "Agreed. Good to clear this up quickly. I was getting worried the competition was over already... I'm glad it's not!", "830674": "Things always happen. Thank you for the fast response.", "829844": "", "826739": "", "849607": "Thanks for messaging.\n", "827330": "Thanks for the fast change!" } }