A few days in, the Public LB is at 0.97 and public code is for 0.95
\nchasing a perfect score of 1.00 ;
\nlooks like doing well in this competition will be about grinding out small metric score differences?
…. and obviously how solid your CV is!!
", "messages": [ { "id": 1304525, "postDate": "2021-05-12T17:11:52.543Z", "content": "A few days in, the Public LB is at 0.97 and public code is for 0.95
\nchasing a perfect score of 1.00 ;
\nlooks like doing well in this competition will be about grinding out small metric score differences?
…. and obviously how solid your CV is!!
", "rawMarkdown": "A few days in, the Public LB is at 0.97 and public code is for 0.95\nchasing a perfect score of 1.00 ; \nlooks like doing well in this competition will be about grinding out small metric score differences?\n\n.... and obviously how solid your CV is!! \n\n", "votes": 3 }, { "id": 1304532, "postDate": "2021-05-12T17:19:32.673Z", "content": "@kmldas @maggiemd - maybe we could persuade Kaggle to show more than 2dp on the LB for this one?
", "rawMarkdown": "@kmldas @maggiemd - maybe we could persuade Kaggle to show more than 2dp on the LB for this one?", "votes": 2, "replies": [ { "id": 1304536, "postDate": "2021-05-12T17:24:21.257Z", "content": "Sure, to enable LB probing. ;)
", "rawMarkdown": "Sure, to enable LB probing. ;)", "votes": 1 }, { "id": 1304607, "postDate": "2021-05-12T18:23:29.020Z", "content": "hi @jbomitchell ,
\n@cpmpml has highlighted the reasons for limiting scores decimal points in another thread \"Limiting the display accuracy is meant to prevent LB probing. I doubt it will be changed. If you want to blame someone for this low digit display then blame me as I am the one who suggested limited display accuracy to Kaggle…\"
\nhttps://www.kaggle.com/c/birdclef-2021/discussion/232913
Also, the results of the last few competitions show that public score is not a great indicator! many including me have been hit with differences in image size and not debugging properly
\nWe should trust our CV…. and Theo Viel's suggestion in the below thread are very helpful: https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/238150
", "rawMarkdown": "hi @jbomitchell , \n@cpmpml has highlighted the reasons for limiting scores decimal points in another thread \"Limiting the display accuracy is meant to prevent LB probing. I doubt it will be changed. If you want to blame someone for this low digit display then blame me as I am the one who suggested limited display accuracy to Kaggle…\"\nhttps://www.kaggle.com/c/birdclef-2021/discussion/232913\n\nAlso, the results of the last few competitions show that public score is not a great indicator! many including me have been hit with differences in image size and not debugging properly\n\nWe should trust our CV.... and Theo Viel's suggestion in the below thread are very helpful: https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/238150 ", "votes": 4 }, { "id": 1304630, "postDate": "2021-05-12T18:42:03.950Z", "content": "I believe that internal validation vs LB correlation, or lack thereof, has been discussed in a number of other places, probably including shopping arcades in China. When 20 people are on 1.00 and the next 150 on 0.99 (which may be soon given the rocket powered start to this competition), we will all be flying blind - apart from the \"Did that submission move me up the queue of Kagglers on identical scores\" metric (and @kmldas on that measure you are currently two places further up the line than me).
\n\"Your submission scored 1.00, which is not an improvement of your best score. Keep trying!\"
", "rawMarkdown": "I believe that internal validation vs LB correlation, or lack thereof, has been discussed in a number of other places, probably including shopping arcades in China. When 20 people are on 1.00 and the next 150 on 0.99 (which may be soon given the rocket powered start to this competition), we will all be flying blind - apart from the \"Did that submission move me up the queue of Kagglers on identical scores\" metric (and @kmldas on that measure you are currently two places further up the line than me).\n\n\"Your submission scored 1.00, which is not an improvement of your best score. Keep trying!\"", "votes": 1 }, { "id": 1304639, "postDate": "2021-05-12T18:48:05.230Z", "content": "\n\n\"Your submission scored 1.00, which is not an improvement of your best score. Keep trying!\"
\n
😄 I can definitely see that.. and hoping to see you at 1.00 @jbomitchell !! My best wishes!!
\nagree! the close competition and lack of Public LB validation with limited decimal points is a challenge. To be honest, I am surprised I got this far 2 days into a competition. And am unsure what more I can do …
\nHowever, unlikely we will get more decimal points so no use wondering about it.
\nBest we may do is focus on CV and not be lured by ensembling which is already a top 10 contender in Public LB…
Also, a similar request here: https://www.kaggle.com/c/seti-breakthrough-listen/discussion/238270
", "rawMarkdown": "Also, a similar request here: https://www.kaggle.com/c/seti-breakthrough-listen/discussion/238270", "votes": 2 }, { "id": 1304739, "postDate": "2021-05-12T20:31:27.403Z", "content": "\n\nWhen 20 people are on 1.00 and the next 150 on 0.99 (which may be soon given the rocket powered start to this competition), we will all be flying blind
\n
Why would this be a problem?
\nUsing feedback from test set is one of the worst practice in real life machine learning. It is a good thing if it is useless here.
", "rawMarkdown": "> When 20 people are on 1.00 and the next 150 on 0.99 (which may be soon given the rocket powered start to this competition), we will all be flying blind \n\nWhy would this be a problem?\n\nUsing feedback from test set is one of the worst practice in real life machine learning. It is a good thing if it is useless here.", "votes": 3 }, { "id": 1304767, "postDate": "2021-05-12T21:13:42.863Z", "content": "[Completely rewritten] What we are being asked to do is essentially to assign every true instance a higher score than any false instance. While AUC is a decent measure for weak predictors, at the level of accuracy we're seeing here, the AUC to 2dp is a very coarse measure of the number of misranked instances. If I were refereeing a paper or thesis that used this metric and precision for predictors like these, I would insist that it was changed.
", "rawMarkdown": "[Completely rewritten] What we are being asked to do is essentially to assign every true instance a higher score than any false instance. While AUC is a decent measure for weak predictors, at the level of accuracy we're seeing here, the AUC to 2dp is a very coarse measure of the number of misranked instances. If I were refereeing a paper or thesis that used this metric and precision for predictors like these, I would insist that it was changed. ", "votes": 2 } ] } ], "comments": [ { "id": 1304532, "author_name": "John Mitchell", "author_url": "", "post_date": "2021-05-12T17:19:32.673000", "content": "@kmldas @maggiemd - maybe we could persuade Kaggle to show more than 2dp on the LB for this one?
", "votes": 2, "replies": [ { "id": 1304536, "author_name": "CPMP", "author_url": "", "post_date": "2021-05-12T17:24:21.257000", "content": "Sure, to enable LB probing. ;)
", "votes": 1, "replies": [] }, { "id": 1304607, "author_name": "Kamal Das", "author_url": "", "post_date": "2021-05-12T18:23:29.020000", "content": "hi @jbomitchell ,
\n@cpmpml has highlighted the reasons for limiting scores decimal points in another thread \"Limiting the display accuracy is meant to prevent LB probing. I doubt it will be changed. If you want to blame someone for this low digit display then blame me as I am the one who suggested limited display accuracy to Kaggle…\"
\nhttps://www.kaggle.com/c/birdclef-2021/discussion/232913
Also, the results of the last few competitions show that public score is not a great indicator! many including me have been hit with differences in image size and not debugging properly
\nWe should trust our CV…. and Theo Viel's suggestion in the below thread are very helpful: https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/238150
", "votes": 4, "replies": [] }, { "id": 1304630, "author_name": "John Mitchell", "author_url": "", "post_date": "2021-05-12T18:42:03.950000", "content": "I believe that internal validation vs LB correlation, or lack thereof, has been discussed in a number of other places, probably including shopping arcades in China. When 20 people are on 1.00 and the next 150 on 0.99 (which may be soon given the rocket powered start to this competition), we will all be flying blind - apart from the \"Did that submission move me up the queue of Kagglers on identical scores\" metric (and @kmldas on that measure you are currently two places further up the line than me).
\n\"Your submission scored 1.00, which is not an improvement of your best score. Keep trying!\"
", "votes": 1, "replies": [] }, { "id": 1304639, "author_name": "Kamal Das", "author_url": "", "post_date": "2021-05-12T18:48:05.230000", "content": "\n\n\"Your submission scored 1.00, which is not an improvement of your best score. Keep trying!\"
\n
😄 I can definitely see that.. and hoping to see you at 1.00 @jbomitchell !! My best wishes!!
\nagree! the close competition and lack of Public LB validation with limited decimal points is a challenge. To be honest, I am surprised I got this far 2 days into a competition. And am unsure what more I can do …
\nHowever, unlikely we will get more decimal points so no use wondering about it.
\nBest we may do is focus on CV and not be lured by ensembling which is already a top 10 contender in Public LB…
Also, a similar request here: https://www.kaggle.com/c/seti-breakthrough-listen/discussion/238270
", "votes": 2, "replies": [] }, { "id": 1304739, "author_name": "CPMP", "author_url": "", "post_date": "2021-05-12T20:31:27.403000", "content": "\n\nWhen 20 people are on 1.00 and the next 150 on 0.99 (which may be soon given the rocket powered start to this competition), we will all be flying blind
\n
Why would this be a problem?
\nUsing feedback from test set is one of the worst practice in real life machine learning. It is a good thing if it is useless here.
", "votes": 3, "replies": [] }, { "id": 1304767, "author_name": "John Mitchell", "author_url": "", "post_date": "2021-05-12T21:13:42.863000", "content": "[Completely rewritten] What we are being asked to do is essentially to assign every true instance a higher score than any false instance. While AUC is a decent measure for weak predictors, at the level of accuracy we're seeing here, the AUC to 2dp is a very coarse measure of the number of misranked instances. If I were refereeing a paper or thesis that used this metric and precision for predictors like these, I would insist that it was changed.
", "votes": 2, "replies": [] } ] } ], "raw_markdown_by_id": { "1304525": "A few days in, the Public LB is at 0.97 and public code is for 0.95\nchasing a perfect score of 1.00 ; \nlooks like doing well in this competition will be about grinding out small metric score differences?\n\n.... and obviously how solid your CV is!! \n\n", "1304532": "@kmldas @maggiemd - maybe we could persuade Kaggle to show more than 2dp on the LB for this one?" } }