{ "id": 195729, "title": "Lyft Track ID Does Not Matter", "url": "/competitions/lyft-motion-prediction-autonomous-vehicles/discussion/195729", "author_name": "", "post_date": "2020-11-07T02:44:53.732530800Z", "votes": 22, "comment_count": 15, "views": 0, "content": "

It seems weird to me that I can just take a public notebook, set all the track_ids to zero and still get the same score !!!

\n

Isn't something wrong with the implementation of the evaluation metric ? No merge is being done internally ? What if I shuffle my submission file ?

\n

# Update 1

\n

And when I shuffle, it leads to a submission error 😨 ? This could jeopardise submissions and best models scores !!!

", "messages": [ { "id": "1071553", "postDate": "11/07/2020 02:44:53", "content": "

It seems weird to me that I can just take a public notebook, set all the track_ids to zero and still get the same score !!!

\n

Isn't something wrong with the implementation of the evaluation metric ? No merge is being done internally ? What if I shuffle my submission file ?

\n

# Update 1

\n

And when I shuffle, it leads to a submission error 😨 ? This could jeopardise submissions and best models scores !!!

", "rawMarkdown": "It seems weird to me that I can just take a [public notebook](https://www.kaggle.com/huanvo/lyft-complete-train-and-prediction-pipeline), set all [the track_ids to zero](https://www.kaggle.com/kneroma/lyft-track-id-does-not-matter) and still get the same score !!!\n\nIsn't something wrong with the implementation of the evaluation metric ? No merge is being done internally ? What if I shuffle my submission file ? \n\n# # Update 1\nAnd when I shuffle, it leads to a submission error 😨 ? This could jeopardise submissions and best models scores !!!", "votes": null }, { "id": "1072807", "postDate": "11/08/2020 17:47:36", "content": "

@iglovikov @philculliton can you enlighten us about this issue ?

", "rawMarkdown": "iglovikov @philculliton can you enlighten us about this issue ?", "votes": null }, { "id": "1073610", "postDate": "11/09/2020 17:43:06", "content": "

Hey, are the notebooks running with default l5kit? I can't reproduce the issue offline using your submission and your code. I'm trying to understand if it's an issue in l5kit or kaggle code itself

", "rawMarkdown": "Hey, are the notebooks running with default l5kit? I can't reproduce the issue offline using your submission and your code. I'm trying to understand if it's an issue in l5kit or kaggle code itself", "votes": null }, { "id": "1073648", "postDate": "11/09/2020 18:40:28", "content": "

You can reproduce everything by forking my notebook and submit it.

", "rawMarkdown": "You can reproduce everything by [forking my notebook](https://www.kaggle.com/kneroma/lyft-track-id-does-not-matter) and submit it.", "votes": null }, { "id": "1073719", "postDate": "11/09/2020 21:12:22", "content": "

I've just heard back from Kaggle. They have implemented our evaluation protocol in a slightly different way. They just check if the two timestamps match while for looping over the submission rows.
\nThis explains why setting all track_id to 0 doesn't affect the results and why submission rows must be sorted to work properly.

\n

In the end I don't think there is any issue really. Not an ideal implementation for sure, but it's not causing any problem to the leaderboard.

", "rawMarkdown": "I've just heard back from Kaggle. They have implemented our evaluation protocol in a slightly different way. They just check if the two timestamps match while for looping over the submission rows. \nThis explains why setting all track_id to 0 doesn't affect the results and why submission rows must be sorted to work properly.\n\nIn the end I don't think there is any issue really. Not an ideal implementation for sure, but it's not causing any problem to the leaderboard.", "votes": null }, { "id": "1076444", "postDate": "11/12/2020 14:37:39", "content": "

I'm not so sure about the fact that this is not problematic.

\n

Not only it would flag many submissions as non correct but also, it opens some gates for private test probing … You may need to pay a special attention to my last statement :)

", "rawMarkdown": "I'm not so sure about the fact that this is not problematic.\n\nNot only it would flag many submissions as non correct but also, it opens some gates for private test probing ... You may need to pay a special attention to my last statement :)", "votes": null }, { "id": "1076455", "postDate": "11/12/2020 14:50:27", "content": "

it will flag a submission as not correct only if you shuffle, which is not something you usually do for the test set anyway. I haven't heard any reports of this (apart from this post), so I don't think a lot of people are doing it. Also, the baseline doesn't shuffle, so everything stemming from them will likely be correct. I may start a post for this anyway :)

\n
\n

it opens some gates for private test probing

\n
\n

can you give me an example of this? I honestly can't find one..

", "rawMarkdown": "it will flag a submission as not correct only if you shuffle, which is not something you usually do for the test set anyway. I haven't heard any reports of this (apart from this post), so I don't think a lot of people are doing it. Also, the baseline doesn't shuffle, so everything stemming from them will likely be correct. I may start a post for this anyway :)\n\n> it opens some gates for private test probing\n\ncan you give me an example of this? I honestly can't find one..", "votes": null }, { "id": "1076646", "postDate": "11/12/2020 18:00:48", "content": "

Yes, shuffling is not common, but we can submit a csv after merging or just after doing many processings and then, if we forget the order, things won't work as expected. But, anyway, if it's Ok for you arganizers, it's Ok for me too.

\n

About using this \"bug\" to probe the LB:

\n

Just select a timestamp, shuffle all the rows for this timestamp and submit : if your score change, then you're on the Public test set, otherwise it's the Private test set. One can use this trick to guess the Private test set rows and deal with them accordingly, unfortunately ☹️.

\n

And thanks for your responsiveness as organizer of this nice competition :)

", "rawMarkdown": "Yes, shuffling is not common, but we can submit a csv after merging or just after doing many processings and then, if we forget the order, things won't work as expected. But, anyway, if it's Ok for you arganizers, it's Ok for me too.\n\nAbout using this \"bug\" to probe the LB:\n\nJust select a timestamp, shuffle all the rows for this timestamp and submit : if your score change, then you're on the Public test set, otherwise it's the Private test set. One can use this trick to guess the Private test set rows and deal with them accordingly, unfortunately ☹️. \n\nAnd thanks for your responsiveness as organizer of this nice competition :)", "votes": null }, { "id": "1076671", "postDate": "11/12/2020 18:20:29", "content": "
\n

Just select a timestamp, shuffle all the rows for this timestamp and submit : if your score change, then you're on the Public test set, otherwise it's the Private test set. One can use this trick to guess the Private test set rows and deal with them accordingly, unfortunately ☹️.

\n
\n

You're right, that's possible :) But given the short time left I don't think that's going to be an issue (also considering the daily limit). I will report this to Vlad and see what he think though!

", "rawMarkdown": "> Just select a timestamp, shuffle all the rows for this timestamp and submit : if your score change, then you're on the Public test set, otherwise it's the Private test set. One can use this trick to guess the Private test set rows and deal with them accordingly, unfortunately ☹️.\n\nYou're right, that's possible :) But given the short time left I don't think that's going to be an issue (also considering the daily limit). I will report this to Vlad and see what he think though!", "votes": null }, { "id": "1076679", "postDate": "11/12/2020 18:29:12", "content": "

This is a good point. It looks like one can probe the LB. We should have avoided this.

\n

But I do not see how this can have an impact on the Leaderboard.

", "rawMarkdown": "This is a good point. It looks like one can probe the LB. We should have avoided this.\n\nBut I do not see how this can have an impact on the Leaderboard.", "votes": null }, { "id": "1076688", "postDate": "11/12/2020 18:41:41", "content": "

You don't have to probe, take a look at my recent post

", "rawMarkdown": "You don't have to probe, take a look at my recent [post](https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/196796)", "votes": null }, { "id": "1076715", "postDate": "11/12/2020 19:22:24", "content": "

Tried to have this conversation two months back. You can easily get private LB using peter solution and know the score. https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/182117

", "rawMarkdown": "Tried to have this conversation two months back. You can easily get private LB using peter solution and know the score. [https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/182117 ](url)", "votes": null }, { "id": "1081341", "postDate": "11/17/2020 02:19:29", "content": "

I guess it has already been probed ;)

", "rawMarkdown": "I guess it has already been probed ;)", "votes": null }, { "id": "1081992", "postDate": "11/17/2020 13:54:51", "content": "

I dont see how you can probe the score of private?

", "rawMarkdown": "I dont see how you can probe the score of private?", "votes": null }, { "id": "1082018", "postDate": "11/17/2020 14:15:45", "content": "

@philippsinger one can set the Private LB as validation set for example, this is not a true probing but it leads to some information leak though :)

", "rawMarkdown": "philippsinger one can set the Private LB as validation set for example, this is not a **true** probing but it leads to some information leak though :)", "votes": null }, { "id": "1082222", "postDate": "11/17/2020 17:59:16", "content": "

I dont think this tells you much as you have still no real access to the frames being scored.

", "rawMarkdown": "I dont think this tells you much as you have still no real access to the frames being scored.", "votes": null } ], "comments": [ { "id": 1072807, "author_name": "kneroma", "author_url": "", "post_date": "11/08/2020 17:47:36", "content": "

@iglovikov @philculliton can you enlighten us about this issue ?

", "votes": null, "replies": [ { "id": 1073610, "author_name": "lucabergamini", "author_url": "", "post_date": "11/09/2020 17:43:06", "content": "

Hey, are the notebooks running with default l5kit? I can't reproduce the issue offline using your submission and your code. I'm trying to understand if it's an issue in l5kit or kaggle code itself

", "votes": null, "replies": [] }, { "id": 1073648, "author_name": "kneroma", "author_url": "", "post_date": "11/09/2020 18:40:28", "content": "

You can reproduce everything by forking my notebook and submit it.

", "votes": null, "replies": [] }, { "id": 1073719, "author_name": "lucabergamini", "author_url": "", "post_date": "11/09/2020 21:12:22", "content": "

I've just heard back from Kaggle. They have implemented our evaluation protocol in a slightly different way. They just check if the two timestamps match while for looping over the submission rows.
\nThis explains why setting all track_id to 0 doesn't affect the results and why submission rows must be sorted to work properly.

\n

In the end I don't think there is any issue really. Not an ideal implementation for sure, but it's not causing any problem to the leaderboard.

", "votes": null, "replies": [] }, { "id": 1076444, "author_name": "kneroma", "author_url": "", "post_date": "11/12/2020 14:37:39", "content": "

I'm not so sure about the fact that this is not problematic.

\n

Not only it would flag many submissions as non correct but also, it opens some gates for private test probing … You may need to pay a special attention to my last statement :)

", "votes": null, "replies": [] }, { "id": 1076455, "author_name": "lucabergamini", "author_url": "", "post_date": "11/12/2020 14:50:27", "content": "

it will flag a submission as not correct only if you shuffle, which is not something you usually do for the test set anyway. I haven't heard any reports of this (apart from this post), so I don't think a lot of people are doing it. Also, the baseline doesn't shuffle, so everything stemming from them will likely be correct. I may start a post for this anyway :)

\n
\n

it opens some gates for private test probing

\n
\n

can you give me an example of this? I honestly can't find one..

", "votes": null, "replies": [] }, { "id": 1076646, "author_name": "kneroma", "author_url": "", "post_date": "11/12/2020 18:00:48", "content": "

Yes, shuffling is not common, but we can submit a csv after merging or just after doing many processings and then, if we forget the order, things won't work as expected. But, anyway, if it's Ok for you arganizers, it's Ok for me too.

\n

About using this \"bug\" to probe the LB:

\n

Just select a timestamp, shuffle all the rows for this timestamp and submit : if your score change, then you're on the Public test set, otherwise it's the Private test set. One can use this trick to guess the Private test set rows and deal with them accordingly, unfortunately ☹️.

\n

And thanks for your responsiveness as organizer of this nice competition :)

", "votes": null, "replies": [] }, { "id": 1076671, "author_name": "lucabergamini", "author_url": "", "post_date": "11/12/2020 18:20:29", "content": "
\n

Just select a timestamp, shuffle all the rows for this timestamp and submit : if your score change, then you're on the Public test set, otherwise it's the Private test set. One can use this trick to guess the Private test set rows and deal with them accordingly, unfortunately ☹️.

\n
\n

You're right, that's possible :) But given the short time left I don't think that's going to be an issue (also considering the daily limit). I will report this to Vlad and see what he think though!

", "votes": null, "replies": [] }, { "id": 1076679, "author_name": "iglovikov", "author_url": "", "post_date": "11/12/2020 18:29:12", "content": "

This is a good point. It looks like one can probe the LB. We should have avoided this.

\n

But I do not see how this can have an impact on the Leaderboard.

", "votes": null, "replies": [] }, { "id": 1076688, "author_name": "pestipeti", "author_url": "", "post_date": "11/12/2020 18:41:41", "content": "

You don't have to probe, take a look at my recent post

", "votes": null, "replies": [] }, { "id": 1076715, "author_name": "deepakrajpurushothaman", "author_url": "", "post_date": "11/12/2020 19:22:24", "content": "

Tried to have this conversation two months back. You can easily get private LB using peter solution and know the score. https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/182117

", "votes": null, "replies": [] }, { "id": 1081341, "author_name": "louis925", "author_url": "", "post_date": "11/17/2020 02:19:29", "content": "

I guess it has already been probed ;)

", "votes": null, "replies": [] }, { "id": 1081992, "author_name": "philippsinger", "author_url": "", "post_date": "11/17/2020 13:54:51", "content": "

I dont see how you can probe the score of private?

", "votes": null, "replies": [] }, { "id": 1082018, "author_name": "kneroma", "author_url": "", "post_date": "11/17/2020 14:15:45", "content": "

@philippsinger one can set the Private LB as validation set for example, this is not a true probing but it leads to some information leak though :)

", "votes": null, "replies": [] }, { "id": 1082222, "author_name": "philippsinger", "author_url": "", "post_date": "11/17/2020 17:59:16", "content": "

I dont think this tells you much as you have still no real access to the frames being scored.

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1071553": "It seems weird to me that I can just take a [public notebook](https://www.kaggle.com/huanvo/lyft-complete-train-and-prediction-pipeline), set all [the track_ids to zero](https://www.kaggle.com/kneroma/lyft-track-id-does-not-matter) and still get the same score !!!\n\nIsn't something wrong with the implementation of the evaluation metric ? No merge is being done internally ? What if I shuffle my submission file ? \n\n# # Update 1\nAnd when I shuffle, it leads to a submission error 😨 ? This could jeopardise submissions and best models scores !!!", "1072807": "iglovikov @philculliton can you enlighten us about this issue ?", "1073610": "Hey, are the notebooks running with default l5kit? I can't reproduce the issue offline using your submission and your code. I'm trying to understand if it's an issue in l5kit or kaggle code itself", "1073648": "You can reproduce everything by [forking my notebook](https://www.kaggle.com/kneroma/lyft-track-id-does-not-matter) and submit it.", "1073719": "I've just heard back from Kaggle. They have implemented our evaluation protocol in a slightly different way. They just check if the two timestamps match while for looping over the submission rows. \nThis explains why setting all track_id to 0 doesn't affect the results and why submission rows must be sorted to work properly.\n\nIn the end I don't think there is any issue really. Not an ideal implementation for sure, but it's not causing any problem to the leaderboard.", "1076444": "I'm not so sure about the fact that this is not problematic.\n\nNot only it would flag many submissions as non correct but also, it opens some gates for private test probing ... You may need to pay a special attention to my last statement :)", "1076455": "it will flag a submission as not correct only if you shuffle, which is not something you usually do for the test set anyway. I haven't heard any reports of this (apart from this post), so I don't think a lot of people are doing it. Also, the baseline doesn't shuffle, so everything stemming from them will likely be correct. I may start a post for this anyway :)\n\n> it opens some gates for private test probing\n\ncan you give me an example of this? I honestly can't find one..", "1076646": "Yes, shuffling is not common, but we can submit a csv after merging or just after doing many processings and then, if we forget the order, things won't work as expected. But, anyway, if it's Ok for you arganizers, it's Ok for me too.\n\nAbout using this \"bug\" to probe the LB:\n\nJust select a timestamp, shuffle all the rows for this timestamp and submit : if your score change, then you're on the Public test set, otherwise it's the Private test set. One can use this trick to guess the Private test set rows and deal with them accordingly, unfortunately ☹️. \n\nAnd thanks for your responsiveness as organizer of this nice competition :)", "1076671": "> Just select a timestamp, shuffle all the rows for this timestamp and submit : if your score change, then you're on the Public test set, otherwise it's the Private test set. One can use this trick to guess the Private test set rows and deal with them accordingly, unfortunately ☹️.\n\nYou're right, that's possible :) But given the short time left I don't think that's going to be an issue (also considering the daily limit). I will report this to Vlad and see what he think though!", "1076679": "This is a good point. It looks like one can probe the LB. We should have avoided this.\n\nBut I do not see how this can have an impact on the Leaderboard.", "1076688": "You don't have to probe, take a look at my recent [post](https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/196796)", "1076715": "Tried to have this conversation two months back. You can easily get private LB using peter solution and know the score. [https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/182117 ](url)", "1081341": "I guess it has already been probed ;)", "1081992": "I dont see how you can probe the score of private?", "1082018": "philippsinger one can set the Private LB as validation set for example, this is not a **true** probing but it leads to some information leak though :)", "1082222": "I dont think this tells you much as you have still no real access to the frames being scored." }, "source": "meta" }