{ "id": 235328, "title": "Simple trick to hide public LB score", "url": "/competitions/indoor-location-navigation/discussion/235328", "author_name": "mamas", "post_date": "2021-04-28T22:55:47.249000", "votes": 25, "comment_count": 11, "views": 0, "content": "
    \n
  1. Identify 1 sample that is in the public LB by changing the prediction one by one.
    2. \n
  2. Set a big value (e.g. 10000) for the floor prediction of this sample and check how much the score increases. I assume the score increases from 10 to 120.
    3. \n
  3. Then, you can hide your best score to all the other participants. If your submission scores 113.22, it means the true score of your submission is 113.22 - 110 = 3.22. However, you only can know your true score is 3.22.
    4. \n
\n

I hope this is useful for the kagglers who want to surprise many participants by a big shake-up🙏

", "messages": [ { "id": 1287311, "postDate": "2021-04-28T22:55:47.250Z", "content": "
    \n
  1. Identify 1 sample that is in the public LB by changing the prediction one by one.
    2. \n
  2. Set a big value (e.g. 10000) for the floor prediction of this sample and check how much the score increases. I assume the score increases from 10 to 120.
    3. \n
  3. Then, you can hide your best score to all the other participants. If your submission scores 113.22, it means the true score of your submission is 113.22 - 110 = 3.22. However, you only can know your true score is 3.22.
    4. \n
\n

I hope this is useful for the kagglers who want to surprise many participants by a big shake-up🙏

", "rawMarkdown": "1. Identify 1 sample that is in the public LB by changing the prediction one by one. \n2. Set a big value (e.g. 10000) for the floor prediction of this sample and check how much the score increases. I assume the score increases from 10 to 120. \n3. Then, you can hide your best score to all the other participants. If your submission scores 113.22, it means the true score of your submission is 113.22 - 110 = 3.22. However, you only can know your true score is 3.22. \n\nI hope this is useful for the kagglers who want to surprise many participants by a big shake-up🙏", "votes": 25 }, { "id": 1288087, "postDate": "2021-04-29T17:03:11.973Z", "content": "

There is a simpler way. Just shift the floor prediction by +1/-1 for all data and the score will go up 15. And on the last day just resubmit with the correct floor prediction. 😆

", "rawMarkdown": "There is a simpler way. Just shift the floor prediction by +1/-1 for all data and the score will go up 15. And on the last day just resubmit with the correct floor prediction. 😆", "votes": 4, "replies": [ { "id": 1288099, "postDate": "2021-04-29T17:19:26.060Z", "content": "

Oh, you are a genius!

", "rawMarkdown": "Oh, you are a genius! ", "votes": 1 }, { "id": 1288189, "postDate": "2021-04-29T18:32:58.847Z", "content": "

I am beginning to wonder if both of you will show a score of near 0 on the last day 😄

", "rawMarkdown": "I am beginning to wonder if both of you will show a score of near 0 on the last day 😄 ", "votes": 1 }, { "id": 1288198, "postDate": "2021-04-29T18:37:57.237Z", "content": "

To be honest, I'm afraid Tom & dott team, who scores 6.2, is hiding a score now and show 0.x on the last day.

", "rawMarkdown": "To be honest, I'm afraid Tom & dott team, who scores 6.2, is hiding a score now and show 0.x on the last day. ", "votes": 6 }, { "id": 1288203, "postDate": "2021-04-29T18:41:32.127Z", "content": "

very possible… we'll know in a couple of weeks!

\n

I have been struggling for weeks and just managed to scape from 4.052 to 4.050
\nneed some magic and goodness to rub off… looking to work more this weekend and have a 3. somethings score!!

", "rawMarkdown": "very possible... we'll know in a couple of weeks!\n\nI have been struggling for weeks and just managed to scape from 4.052 to 4.050\nneed some magic and goodness to rub off... looking to work more this weekend and have a 3. somethings score!! ", "votes": 1 }, { "id": 1294056, "postDate": "2021-05-05T11:39:17.893Z", "content": "

HI @mamasinkgs and @jiweiliu !!

\n

What are you doing in terms of checking for generalisation to avoid overfitting/shakeup?

\n

I have done so many post processing that I have no clue on my CV …. while I managed to come below 4 to a 3. somethings score, I am worried it may be over fitting to Public LB
\nwhat are you doing to ensure that you are on right side of shakeup?

\n

Thank you for kind suggestion… have worked very hard on this competition and do not want to be shaken away in a few week when the final results are out… Thanks!

", "rawMarkdown": "HI @mamasinkgs and @jiweiliu !!\n\nWhat are you doing in terms of checking for generalisation to avoid overfitting/shakeup?\n\nI have done so many post processing that I have no clue on my CV .... while I managed to come below 4 to a 3. somethings score, I am worried it may be over fitting to Public LB\nwhat are you doing to ensure that you are on right side of shakeup?\n\nThank you for kind suggestion... have worked very hard on this competition and do not want to be shaken away in a few week when the final results are out... Thanks!" }, { "id": 1294225, "postDate": "2021-05-05T13:41:55.440Z", "content": "

I think it is very important to check whether the floor prediciton is correct to avoid shakeup. For the waypoints prediction, it is very important to have a stable CV. How about trying your postprocessing to the validation prediction as well as the test prediction?

", "rawMarkdown": "I think it is very important to check whether the floor prediciton is correct to avoid shakeup. For the waypoints prediction, it is very important to have a stable CV. How about trying your postprocessing to the validation prediction as well as the test prediction? ", "votes": 1 }, { "id": 1294246, "postDate": "2021-05-05T13:55:06.273Z", "content": "

Thanks @mamasinkgs … for floor prediction mine is similar to @nigelhenry's Simple 99% Accurate Floor Model.. though I have not tested it as you suggested in your other post….

\n

Let me try to use postprocessing to the validation prediction as well as the test prediction… my entire post process takes more than a couple of hours. will need to see how I can reduce the time…and make my code more efficient!

", "rawMarkdown": "Thanks @mamasinkgs ... for floor prediction mine is similar to @nigelhenry's Simple 99% Accurate Floor Model.. though I have not tested it as you suggested in your other post....\n\nLet me try to use postprocessing to the validation prediction as well as the test prediction... my entire post process takes more than a couple of hours. will need to see how I can reduce the time...and make my code more efficient!", "votes": 1 } ] }, { "id": 1294328, "postDate": "2021-05-05T14:54:51.060Z", "content": "

Clearly this approach makes sandbagging possible to pretty much any credible degree. It's also possible (though harder) to \"reverse sandbag\" by using LB probing to overfit some public LB paths. I'm not entirely sure that I see much benefit in doing either, and decided that I was better off with an honest estimate of how strong or weak my predictions are.

", "rawMarkdown": "Clearly this approach makes sandbagging possible to pretty much any credible degree. It's also possible (though harder) to \"reverse sandbag\" by using LB probing to overfit some public LB paths. I'm not entirely sure that I see much benefit in doing either, and decided that I was better off with an honest estimate of how strong or weak my predictions are. ", "votes": 1, "replies": [ { "id": 1294362, "postDate": "2021-05-05T15:21:31.450Z", "content": "

Right, we can get much benefit by this approach. So it's just a joke, though I'm somewhat afraid Tom & dott team is doing this 😨

", "rawMarkdown": "Right, we can get much benefit by this approach. So it's just a joke, though I'm somewhat afraid Tom & dott team is doing this 😨", "votes": 6 }, { "id": 1301561, "postDate": "2021-05-11T06:24:53.103Z", "rawMarkdown": "", "isDeleted": true } ] } ], "comments": [ { "id": 1288087, "author_name": "Jiwei Liu", "author_url": "", "post_date": "2021-04-29T17:03:11.973000", "content": "

There is a simpler way. Just shift the floor prediction by +1/-1 for all data and the score will go up 15. And on the last day just resubmit with the correct floor prediction. 😆

", "votes": 4, "replies": [ { "id": 1288099, "author_name": "mamas", "author_url": "", "post_date": "2021-04-29T17:19:26.060000", "content": "

Oh, you are a genius!

", "votes": 1, "replies": [] }, { "id": 1288189, "author_name": "Kamal Das", "author_url": "", "post_date": "2021-04-29T18:32:58.847000", "content": "

I am beginning to wonder if both of you will show a score of near 0 on the last day 😄

", "votes": 1, "replies": [] }, { "id": 1288198, "author_name": "mamas", "author_url": "", "post_date": "2021-04-29T18:37:57.237000", "content": "

To be honest, I'm afraid Tom & dott team, who scores 6.2, is hiding a score now and show 0.x on the last day.

", "votes": 6, "replies": [] }, { "id": 1288203, "author_name": "Kamal Das", "author_url": "", "post_date": "2021-04-29T18:41:32.127000", "content": "

very possible… we'll know in a couple of weeks!

\n

I have been struggling for weeks and just managed to scape from 4.052 to 4.050
\nneed some magic and goodness to rub off… looking to work more this weekend and have a 3. somethings score!!

", "votes": 1, "replies": [] }, { "id": 1294056, "author_name": "Kamal Das", "author_url": "", "post_date": "2021-05-05T11:39:17.893000", "content": "

HI @mamasinkgs and @jiweiliu !!

\n

What are you doing in terms of checking for generalisation to avoid overfitting/shakeup?

\n

I have done so many post processing that I have no clue on my CV …. while I managed to come below 4 to a 3. somethings score, I am worried it may be over fitting to Public LB
\nwhat are you doing to ensure that you are on right side of shakeup?

\n

Thank you for kind suggestion… have worked very hard on this competition and do not want to be shaken away in a few week when the final results are out… Thanks!

", "votes": 0, "replies": [] }, { "id": 1294225, "author_name": "mamas", "author_url": "", "post_date": "2021-05-05T13:41:55.440000", "content": "

I think it is very important to check whether the floor prediciton is correct to avoid shakeup. For the waypoints prediction, it is very important to have a stable CV. How about trying your postprocessing to the validation prediction as well as the test prediction?

", "votes": 1, "replies": [] }, { "id": 1294246, "author_name": "Kamal Das", "author_url": "", "post_date": "2021-05-05T13:55:06.273000", "content": "

Thanks @mamasinkgs … for floor prediction mine is similar to @nigelhenry's Simple 99% Accurate Floor Model.. though I have not tested it as you suggested in your other post….

\n

Let me try to use postprocessing to the validation prediction as well as the test prediction… my entire post process takes more than a couple of hours. will need to see how I can reduce the time…and make my code more efficient!

", "votes": 1, "replies": [] } ] }, { "id": 1294328, "author_name": "John Mitchell", "author_url": "", "post_date": "2021-05-05T14:54:51.060000", "content": "

Clearly this approach makes sandbagging possible to pretty much any credible degree. It's also possible (though harder) to \"reverse sandbag\" by using LB probing to overfit some public LB paths. I'm not entirely sure that I see much benefit in doing either, and decided that I was better off with an honest estimate of how strong or weak my predictions are.

", "votes": 1, "replies": [ { "id": 1294362, "author_name": "mamas", "author_url": "", "post_date": "2021-05-05T15:21:31.450000", "content": "

Right, we can get much benefit by this approach. So it's just a joke, though I'm somewhat afraid Tom & dott team is doing this 😨

", "votes": 6, "replies": [] }, { "id": 1301561, "author_name": "", "author_url": "", "post_date": "2021-05-11T06:24:53.103000", "content": "", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1287311": "1. Identify 1 sample that is in the public LB by changing the prediction one by one. \n2. Set a big value (e.g. 10000) for the floor prediction of this sample and check how much the score increases. I assume the score increases from 10 to 120. \n3. Then, you can hide your best score to all the other participants. If your submission scores 113.22, it means the true score of your submission is 113.22 - 110 = 3.22. However, you only can know your true score is 3.22. \n\nI hope this is useful for the kagglers who want to surprise many participants by a big shake-up🙏", "1288087": "There is a simpler way. Just shift the floor prediction by +1/-1 for all data and the score will go up 15. And on the last day just resubmit with the correct floor prediction. 😆", "1294328": "Clearly this approach makes sandbagging possible to pretty much any credible degree. It's also possible (though harder) to \"reverse sandbag\" by using LB probing to overfit some public LB paths. I'm not entirely sure that I see much benefit in doing either, and decided that I was better off with an honest estimate of how strong or weak my predictions are. " } }