{ "id": 182117, "title": "Whats the meaning of evaluation against 50% test set on LB?", "url": "/competitions/lyft-motion-prediction-autonomous-vehicles/discussion/182117", "author_name": "", "post_date": "2020-09-11T11:24:24.245581300Z", "votes": 1, "comment_count": 3, "views": 0, "content": "

If you see the evaluation criteria. It says that Public LB is based on 50% of test set. In my mind this has two possibility:
\n1) when we submit our predictions only 25 frames for each of the hypothesis is being evaluated. That means future path prediction/frame prediction for the first 2.5 sec.
\n2) Another possibility is that there is another test set for which our model will be evaluated against. Since this is a code competition and run time of most of the model inference that i can imagine might fall well within kaggle limit/competation notebook limit of 9hrs.

\n

Which one of these could mean 50% test set evaluation?

\n

In whatever the case both seem quite challenging. In first case, any prediction from 2.5 sec to 5 sec(the last 25 frames of each of the hypothesis) will be more error prone. So private leader board might end up looking different. In case of second possibility, I fear we might tune our solution more for existing ground truth.

", "messages": [ { "id": "1006564", "postDate": "09/11/2020 11:24:24", "content": "

If you see the evaluation criteria. It says that Public LB is based on 50% of test set. In my mind this has two possibility:
\n1) when we submit our predictions only 25 frames for each of the hypothesis is being evaluated. That means future path prediction/frame prediction for the first 2.5 sec.
\n2) Another possibility is that there is another test set for which our model will be evaluated against. Since this is a code competition and run time of most of the model inference that i can imagine might fall well within kaggle limit/competation notebook limit of 9hrs.

\n

Which one of these could mean 50% test set evaluation?

\n

In whatever the case both seem quite challenging. In first case, any prediction from 2.5 sec to 5 sec(the last 25 frames of each of the hypothesis) will be more error prone. So private leader board might end up looking different. In case of second possibility, I fear we might tune our solution more for existing ground truth.

", "rawMarkdown": "If you see the evaluation criteria. It says that Public LB is based on 50% of test set. In my mind this has two possibility: \n1) when we submit our predictions only 25 frames for each of the hypothesis is being evaluated. That means future path prediction/frame prediction for the first 2.5 sec.\n2) Another possibility is that there is another test set for which our model will be evaluated against. Since this is a code competition and run time of most of the model inference that i can imagine might fall well within kaggle limit/competation notebook limit of 9hrs. \n\nWhich one of these could mean 50% test set evaluation?\n\nIn whatever the case both seem quite challenging. In first case, any prediction from 2.5 sec to 5 sec(the last 25 frames of each of the hypothesis) will be more error prone. So private leader board might end up looking different. In case of second possibility, I fear we might tune our solution more for existing ground truth.", "votes": null }, { "id": "1007033", "postDate": "09/11/2020 18:33:56", "content": "

I am quite certain, that 50% of the scenes are evaluated for public LB and the remaining 50% for the private LB. That means all 50 future frames (they will not be splitted).

\n

The hosts already said that it's not really a code competition and we are free to do inference locally and only upload the resulting csv:
\nhttps://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/177912#989502

", "rawMarkdown": "I am quite certain, that 50% of the scenes are evaluated for public LB and the remaining 50% for the private LB. That means all 50 future frames (they will not be splitted). \n\nThe hosts already said that it's not really a code competition and we are free to do inference locally and only upload the resulting csv: \nhttps://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/177912#989502", "votes": null }, { "id": "1007127", "postDate": "09/11/2020 20:32:17", "content": "

Oh this is third possibility. So you mean they are making Division for the samples 71122/2 on the rows and not (305-5)/2 on the columns. This will prolly keep the LB same.

", "rawMarkdown": "Oh this is third possibility. So you mean they are making Division for the samples 71122/2 on the rows and not (305-5)/2 on the columns. This will prolly keep the LB same.", "votes": null }, { "id": "1007179", "postDate": "09/11/2020 22:03:22", "content": "

yes, most likely the split is done by row, not by column (of the csv).

", "rawMarkdown": "yes, most likely the split is done by row, not by column (of the csv).", "votes": null } ], "comments": [ { "id": 1007033, "author_name": "ilu000", "author_url": "", "post_date": "09/11/2020 18:33:56", "content": "

I am quite certain, that 50% of the scenes are evaluated for public LB and the remaining 50% for the private LB. That means all 50 future frames (they will not be splitted).

\n

The hosts already said that it's not really a code competition and we are free to do inference locally and only upload the resulting csv:
\nhttps://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/177912#989502

", "votes": null, "replies": [ { "id": 1007127, "author_name": "deepakrajpurushothaman", "author_url": "", "post_date": "09/11/2020 20:32:17", "content": "

Oh this is third possibility. So you mean they are making Division for the samples 71122/2 on the rows and not (305-5)/2 on the columns. This will prolly keep the LB same.

", "votes": null, "replies": [] }, { "id": 1007179, "author_name": "ilu000", "author_url": "", "post_date": "09/11/2020 22:03:22", "content": "

yes, most likely the split is done by row, not by column (of the csv).

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1006564": "If you see the evaluation criteria. It says that Public LB is based on 50% of test set. In my mind this has two possibility: \n1) when we submit our predictions only 25 frames for each of the hypothesis is being evaluated. That means future path prediction/frame prediction for the first 2.5 sec.\n2) Another possibility is that there is another test set for which our model will be evaluated against. Since this is a code competition and run time of most of the model inference that i can imagine might fall well within kaggle limit/competation notebook limit of 9hrs. \n\nWhich one of these could mean 50% test set evaluation?\n\nIn whatever the case both seem quite challenging. In first case, any prediction from 2.5 sec to 5 sec(the last 25 frames of each of the hypothesis) will be more error prone. So private leader board might end up looking different. In case of second possibility, I fear we might tune our solution more for existing ground truth.", "1007033": "I am quite certain, that 50% of the scenes are evaluated for public LB and the remaining 50% for the private LB. That means all 50 future frames (they will not be splitted). \n\nThe hosts already said that it's not really a code competition and we are free to do inference locally and only upload the resulting csv: \nhttps://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/177912#989502", "1007127": "Oh this is third possibility. So you mean they are making Division for the samples 71122/2 on the rows and not (305-5)/2 on the columns. This will prolly keep the LB same.", "1007179": "yes, most likely the split is done by row, not by column (of the csv)." }, "source": "meta" }