{ "id": 180931, "title": "How to ensemble multi-trajectory predictions?", "url": "/competitions/lyft-motion-prediction-autonomous-vehicles/discussion/180931", "author_name": "", "post_date": "2020-09-06T23:22:37.295236500Z", "votes": 19, "comment_count": 21, "views": 0, "content": "

We can predict up to 3 trajectories with the confidence for each agents in this competition.
\nI could easily achieve score < 45 using the simple baseline model which predicts multi-trajectory with confidence (kernel published).

\n

Now next question arises: how to ensemble predictions when we train more than 2 models?
\nThis may be a key point for this competition to make better predictions.

", "messages": [ { "id": "1000926", "postDate": "09/06/2020 23:22:37", "content": "

We can predict up to 3 trajectories with the confidence for each agents in this competition.
\nI could easily achieve score < 45 using the simple baseline model which predicts multi-trajectory with confidence (kernel published).

\n

Now next question arises: how to ensemble predictions when we train more than 2 models?
\nThis may be a key point for this competition to make better predictions.

", "rawMarkdown": "We can predict up to 3 trajectories with the confidence for each agents in this competition.\nI could easily achieve score < 45 using the simple baseline model which predicts multi-trajectory with confidence ([kernel](https://www.kaggle.com/corochann/lyft-training-with-multi-mode-confidence) published).\n\nNow next question arises: how to ensemble predictions when we train more than 2 models?\nThis may be a key point for this competition to make better predictions.", "votes": null }, { "id": "1000944", "postDate": "09/06/2020 23:53:03", "content": "

I actually wanted to ask about this. I have seen few ensemble notebooks. And also submission.csv of these And I don’t actually understand these notebooks. Isn’t the objective of the algorithm to predict different confidence score for the trajectory? But what I see is, some people are assigning the confidence score themselves to ‚confs‘. If you take a look the submission.csv the confs are equals, one example I can remember is 0.5, 0.3 and 0.2. and these confs are same for all time stamps. I wonder what is the use of predictions if confidence score is assigned manually ? This is my thinking.

", "rawMarkdown": "I actually wanted to ask about this. I have seen few ensemble notebooks. And also submission.csv of these And I don’t actually understand these notebooks. Isn’t the objective of the algorithm to predict different confidence score for the trajectory? But what I see is, some people are assigning the confidence score themselves to ‚confs‘. If you take a look the submission.csv the confs are equals, one example I can remember is 0.5, 0.3 and 0.2. and these confs are same for all time stamps. I wonder what is the use of predictions if confidence score is assigned manually ? This is my thinking.", "votes": null }, { "id": "1000947", "postDate": "09/07/2020 00:01:49", "content": "

In my understanding current existing public kernels predict only 1 trajectory with confidence 1 (1.0, 0.0 0.0).

\n

Below ensembling kernel just adopt each model's single trajectory predictions as each trajectory and manually tune \"confs\" as ensemble hyper parameters:

\n\n

Now I feel, we are start understanding that we can actually build a single cnn which outputs multiple trajectory predictions.
\nIn that case how to ensemble multiple model's predictions where each model predicts multiple trajectory, is not clear.

", "rawMarkdown": "In my understanding current existing public kernels **predict only 1 trajectory with confidence 1 (1.0, 0.0 0.0)**.\n\nBelow ensembling kernel just adopt each model's single trajectory predictions as each trajectory and manually tune \"confs\" as ensemble hyper parameters:\n - https://www.kaggle.com/mekhdigakhramanian/lyft-eda-ensemble-top-lb\n - https://www.kaggle.com/tuckerarrants/lyft-ensembling-raster-sizes\n - https://www.kaggle.com/paulorzp/multi-mode-models-ensemble\n\nNow I feel, we are start understanding that we can actually build a single cnn which outputs multiple trajectory predictions. \nIn that case how to ensemble multiple model's predictions where each model predicts multiple trajectory, is not clear.", "votes": null }, { "id": "1000954", "postDate": "09/07/2020 00:19:19", "content": "

Yes you can have 1.0 0.0 0.0 because its uni modal prediction. Here we actually have to make three prediction, with unimodal you predict only trajectory and coordinates x001/y001 to x050/y050. All other remains value get set to zero for unimodal prediction.
\nBut a constant 0.5, 0.3 and 0.2 or 0.43,0.27, 0.3 doesnt feel right for me. or someone else should make me understand.

", "rawMarkdown": "Yes you can have 1.0 0.0 0.0 because its uni modal prediction. Here we actually have to make three prediction, with unimodal you predict only trajectory and coordinates x001/y001 to x050/y050. All other remains value get set to zero for unimodal prediction. \nBut a constant 0.5, 0.3 and 0.2 or 0.43,0.27, 0.3 doesnt feel right for me. or someone else should make me understand.", "votes": null }, { "id": "1000957", "postDate": "09/07/2020 00:23:29", "content": "

So far in the public kernel, I guess these are manually adjusted.

\n

But I think it's possible to \"optimize\" this parameter too, by preparing the prediction for \"validation dataset\" for each model's single trajectory prediction and optimize these parameters to get best score on this validation dataset.

", "rawMarkdown": "So far in the public kernel, I guess these are manually adjusted.\n\nBut I think it's possible to \"optimize\" this parameter too, by preparing the prediction for \"validation dataset\" for each model's single trajectory prediction and optimize these parameters to get best score on this validation dataset.", "votes": null }, { "id": "1001005", "postDate": "09/07/2020 02:17:37", "content": "

Maybe we can select the three with the highest confidence from all the trajectories of different models, and then normalize their confidence.

", "rawMarkdown": "Maybe we can select the three with the highest confidence from all the trajectories of different models, and then normalize their confidence.", "votes": null }, { "id": "1001026", "postDate": "09/07/2020 02:51:13", "content": "

I think it does not work well to include \"minor\" trajectory.

\n

Selected Top-3 trajectory may be a collapsed trajectory that goes same direction.

", "rawMarkdown": "I think it does not work well to include \"minor\" trajectory.\n\nSelected Top-3 trajectory may be a collapsed trajectory that goes same direction.", "votes": null }, { "id": "1001032", "postDate": "09/07/2020 02:57:11", "content": "

You're right.

", "rawMarkdown": "You're right.", "votes": null }, { "id": "1001040", "postDate": "09/07/2020 03:08:41", "content": "

@corochann I saw your recent commit with muti pred confidence just now. That looks good, a 3 * (X,Y,Confs) output.

", "rawMarkdown": "corochann I saw your recent commit with muti pred confidence just now. That looks good, a 3 * (X,Y,Confs) output.", "votes": null }, { "id": "1001600", "postDate": "09/07/2020 12:32:55", "content": "

Thank you for checking update. Yes I surprised with the scores. I haven't done any parameter tuning yet and that result was from a single model :)

", "rawMarkdown": "Thank you for checking update. Yes I surprised with the scores. I haven't done any parameter tuning yet and that result was from a single model :)", "votes": null }, { "id": "1004657", "postDate": "09/09/2020 22:46:27", "content": "

After training exactly using your notebook( same parameters) for 20000 iterations on kaggle lift dataset, My Lb score is 191 instead of 49. Is there something missing from your training notebook? or are you training on full train dataset available on lyft website instead of kaggle one?

", "rawMarkdown": "After training exactly using your notebook( same parameters) for 20000 iterations on kaggle lift dataset, My Lb score is 191 instead of 49. Is there something missing from your training notebook? or are you training on full train dataset available on lyft website instead of kaggle one?", "votes": null }, { "id": "1004661", "postDate": "09/09/2020 22:55:18", "content": "

Did you set debug=False to use train.zarr instead of sample.zarr?
\nMy current score comes from using train.zarr for about 2 epochs I guess.

", "rawMarkdown": "Did you set `debug=False` to use `train.zarr` instead of `sample.zarr`?\nMy current score comes from using `train.zarr` for about 2 epochs I guess.", "votes": null }, { "id": "1053172", "postDate": "10/18/2020 16:47:37", "content": "

is there any update on ensembling of multi trajectory predictions?

", "rawMarkdown": "is there any update on ensembling of multi trajectory predictions?", "votes": null }, { "id": "1053349", "postDate": "10/18/2020 21:54:20", "content": "

@corochann But 3 confidence ( sum of all 3 = 1 ).. we are prediction 3 trajectory of next 50 steps ?
\nx001/y001 to x050/y050 -- 1st trajectory
\nx101/y101 to x150/y150 -- 2nd trajectory
\nx201/y201 to x250/y250 -- 3rd trajectory

\n

Need to understand what are these x250/y250 predictions ?

", "rawMarkdown": "corochann But 3 confidence ( sum of all 3 = 1 ).. we are prediction 3 trajectory of next 50 steps ?\nx001/y001 to x050/y050 -- 1st trajectory\nx101/y101 to x150/y150 -- 2nd trajectory\nx201/y201 to x250/y250 -- 3rd trajectory\n\nNeed to understand what are these x250/y250 predictions ?", "votes": null }, { "id": "1053358", "postDate": "10/18/2020 22:28:40", "content": "

Yes we are predicting 3 trajectory of next 50 steps.

", "rawMarkdown": "Yes we are predicting 3 trajectory of next 50 steps.", "votes": null }, { "id": "1053362", "postDate": "10/18/2020 22:38:31", "content": "

But what are values for x051/y051 to x100/y100 and x151/y151 to x200/y200 ? not able understand what are these values represent ? @corochann

", "rawMarkdown": "But what are values for x051/y051 to x100/y100 and x151/y151 to x200/y200 ? not able understand what are these values represent ? @corochann", "votes": null }, { "id": "1053363", "postDate": "10/18/2020 22:49:35", "content": "

Does it exist in the submission.csv file? @seshurajup

", "rawMarkdown": "Does it exist in the submission.csv file? @seshurajup", "votes": null }, { "id": "1053419", "postDate": "10/19/2020 01:37:16", "content": "

Now i am clear. Thanks @corochann i miss understood with broken sequence

", "rawMarkdown": "Now i am clear. Thanks @corochann i miss understood with broken sequence", "votes": null }, { "id": "1053473", "postDate": "10/19/2020 03:08:41", "content": "

I noticed same discussion happed here: https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/177408#986337

", "rawMarkdown": "I noticed same discussion happed here: https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/177408#986337", "votes": null }, { "id": "1053717", "postDate": "10/19/2020 09:12:30", "content": "

@corochann Did you train on the entire train.zarr (22M samples) or a subset of the kaggle training data?

", "rawMarkdown": "corochann Did you train on the entire train.zarr (22M samples) or a subset of the kaggle training data?", "votes": null }, { "id": "1053724", "postDate": "10/19/2020 09:24:41", "content": "

I'm trying both cases.

", "rawMarkdown": "I'm trying both cases.", "votes": null }, { "id": "1055430", "postDate": "10/20/2020 19:16:37", "content": "

@corochann Hey. Trying to reach you guys (the whole team) via email. Could you check your letters from kaggle?

", "rawMarkdown": "corochann Hey. Trying to reach you guys (the whole team) via email. Could you check your letters from kaggle?", "votes": null } ], "comments": [ { "id": 1000944, "author_name": "deepakrajpurushothaman", "author_url": "", "post_date": "09/06/2020 23:53:03", "content": "

I actually wanted to ask about this. I have seen few ensemble notebooks. And also submission.csv of these And I don’t actually understand these notebooks. Isn’t the objective of the algorithm to predict different confidence score for the trajectory? But what I see is, some people are assigning the confidence score themselves to ‚confs‘. If you take a look the submission.csv the confs are equals, one example I can remember is 0.5, 0.3 and 0.2. and these confs are same for all time stamps. I wonder what is the use of predictions if confidence score is assigned manually ? This is my thinking.

", "votes": null, "replies": [ { "id": 1000947, "author_name": "corochann", "author_url": "", "post_date": "09/07/2020 00:01:49", "content": "

In my understanding current existing public kernels predict only 1 trajectory with confidence 1 (1.0, 0.0 0.0).

\n

Below ensembling kernel just adopt each model's single trajectory predictions as each trajectory and manually tune \"confs\" as ensemble hyper parameters:

\n\n

Now I feel, we are start understanding that we can actually build a single cnn which outputs multiple trajectory predictions.
\nIn that case how to ensemble multiple model's predictions where each model predicts multiple trajectory, is not clear.

", "votes": null, "replies": [] }, { "id": 1000954, "author_name": "deepakrajpurushothaman", "author_url": "", "post_date": "09/07/2020 00:19:19", "content": "

Yes you can have 1.0 0.0 0.0 because its uni modal prediction. Here we actually have to make three prediction, with unimodal you predict only trajectory and coordinates x001/y001 to x050/y050. All other remains value get set to zero for unimodal prediction.
\nBut a constant 0.5, 0.3 and 0.2 or 0.43,0.27, 0.3 doesnt feel right for me. or someone else should make me understand.

", "votes": null, "replies": [] }, { "id": 1000957, "author_name": "corochann", "author_url": "", "post_date": "09/07/2020 00:23:29", "content": "

So far in the public kernel, I guess these are manually adjusted.

\n

But I think it's possible to \"optimize\" this parameter too, by preparing the prediction for \"validation dataset\" for each model's single trajectory prediction and optimize these parameters to get best score on this validation dataset.

", "votes": null, "replies": [] }, { "id": 1053349, "author_name": "seshurajup", "author_url": "", "post_date": "10/18/2020 21:54:20", "content": "

@corochann But 3 confidence ( sum of all 3 = 1 ).. we are prediction 3 trajectory of next 50 steps ?
\nx001/y001 to x050/y050 -- 1st trajectory
\nx101/y101 to x150/y150 -- 2nd trajectory
\nx201/y201 to x250/y250 -- 3rd trajectory

\n

Need to understand what are these x250/y250 predictions ?

", "votes": null, "replies": [] }, { "id": 1053358, "author_name": "corochann", "author_url": "", "post_date": "10/18/2020 22:28:40", "content": "

Yes we are predicting 3 trajectory of next 50 steps.

", "votes": null, "replies": [] }, { "id": 1053362, "author_name": "seshurajup", "author_url": "", "post_date": "10/18/2020 22:38:31", "content": "

But what are values for x051/y051 to x100/y100 and x151/y151 to x200/y200 ? not able understand what are these values represent ? @corochann

", "votes": null, "replies": [] }, { "id": 1053363, "author_name": "corochann", "author_url": "", "post_date": "10/18/2020 22:49:35", "content": "

Does it exist in the submission.csv file? @seshurajup

", "votes": null, "replies": [] }, { "id": 1053419, "author_name": "seshurajup", "author_url": "", "post_date": "10/19/2020 01:37:16", "content": "

Now i am clear. Thanks @corochann i miss understood with broken sequence

", "votes": null, "replies": [] }, { "id": 1053473, "author_name": "corochann", "author_url": "", "post_date": "10/19/2020 03:08:41", "content": "

I noticed same discussion happed here: https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/177408#986337

", "votes": null, "replies": [] } ] }, { "id": 1001005, "author_name": "zzy990106", "author_url": "", "post_date": "09/07/2020 02:17:37", "content": "

Maybe we can select the three with the highest confidence from all the trajectories of different models, and then normalize their confidence.

", "votes": null, "replies": [ { "id": 1001026, "author_name": "corochann", "author_url": "", "post_date": "09/07/2020 02:51:13", "content": "

I think it does not work well to include \"minor\" trajectory.

\n

Selected Top-3 trajectory may be a collapsed trajectory that goes same direction.

", "votes": null, "replies": [] }, { "id": 1001032, "author_name": "zzy990106", "author_url": "", "post_date": "09/07/2020 02:57:11", "content": "

You're right.

", "votes": null, "replies": [] } ] }, { "id": 1001040, "author_name": "deepakrajpurushothaman", "author_url": "", "post_date": "09/07/2020 03:08:41", "content": "

@corochann I saw your recent commit with muti pred confidence just now. That looks good, a 3 * (X,Y,Confs) output.

", "votes": null, "replies": [ { "id": 1001600, "author_name": "corochann", "author_url": "", "post_date": "09/07/2020 12:32:55", "content": "

Thank you for checking update. Yes I surprised with the scores. I haven't done any parameter tuning yet and that result was from a single model :)

", "votes": null, "replies": [] } ] }, { "id": 1004657, "author_name": "sujaydk", "author_url": "", "post_date": "09/09/2020 22:46:27", "content": "

After training exactly using your notebook( same parameters) for 20000 iterations on kaggle lift dataset, My Lb score is 191 instead of 49. Is there something missing from your training notebook? or are you training on full train dataset available on lyft website instead of kaggle one?

", "votes": null, "replies": [ { "id": 1004661, "author_name": "corochann", "author_url": "", "post_date": "09/09/2020 22:55:18", "content": "

Did you set debug=False to use train.zarr instead of sample.zarr?
\nMy current score comes from using train.zarr for about 2 epochs I guess.

", "votes": null, "replies": [ { "id": 1053717, "author_name": "suryajrrafl", "author_url": "", "post_date": "10/19/2020 09:12:30", "content": "

@corochann Did you train on the entire train.zarr (22M samples) or a subset of the kaggle training data?

", "votes": null, "replies": [] }, { "id": 1053724, "author_name": "corochann", "author_url": "", "post_date": "10/19/2020 09:24:41", "content": "

I'm trying both cases.

", "votes": null, "replies": [ { "id": 1055430, "author_name": "ivanpan", "author_url": "", "post_date": "10/20/2020 19:16:37", "content": "

@corochann Hey. Trying to reach you guys (the whole team) via email. Could you check your letters from kaggle?

", "votes": null, "replies": [] } ] } ] } ] }, { "id": 1053172, "author_name": "sakshamaggarwal", "author_url": "", "post_date": "10/18/2020 16:47:37", "content": "

is there any update on ensembling of multi trajectory predictions?

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1000926": "We can predict up to 3 trajectories with the confidence for each agents in this competition.\nI could easily achieve score < 45 using the simple baseline model which predicts multi-trajectory with confidence ([kernel](https://www.kaggle.com/corochann/lyft-training-with-multi-mode-confidence) published).\n\nNow next question arises: how to ensemble predictions when we train more than 2 models?\nThis may be a key point for this competition to make better predictions.", "1000944": "I actually wanted to ask about this. I have seen few ensemble notebooks. And also submission.csv of these And I don’t actually understand these notebooks. Isn’t the objective of the algorithm to predict different confidence score for the trajectory? But what I see is, some people are assigning the confidence score themselves to ‚confs‘. If you take a look the submission.csv the confs are equals, one example I can remember is 0.5, 0.3 and 0.2. and these confs are same for all time stamps. I wonder what is the use of predictions if confidence score is assigned manually ? This is my thinking.", "1000947": "In my understanding current existing public kernels **predict only 1 trajectory with confidence 1 (1.0, 0.0 0.0)**.\n\nBelow ensembling kernel just adopt each model's single trajectory predictions as each trajectory and manually tune \"confs\" as ensemble hyper parameters:\n - https://www.kaggle.com/mekhdigakhramanian/lyft-eda-ensemble-top-lb\n - https://www.kaggle.com/tuckerarrants/lyft-ensembling-raster-sizes\n - https://www.kaggle.com/paulorzp/multi-mode-models-ensemble\n\nNow I feel, we are start understanding that we can actually build a single cnn which outputs multiple trajectory predictions. \nIn that case how to ensemble multiple model's predictions where each model predicts multiple trajectory, is not clear.", "1000954": "Yes you can have 1.0 0.0 0.0 because its uni modal prediction. Here we actually have to make three prediction, with unimodal you predict only trajectory and coordinates x001/y001 to x050/y050. All other remains value get set to zero for unimodal prediction. \nBut a constant 0.5, 0.3 and 0.2 or 0.43,0.27, 0.3 doesnt feel right for me. or someone else should make me understand.", "1000957": "So far in the public kernel, I guess these are manually adjusted.\n\nBut I think it's possible to \"optimize\" this parameter too, by preparing the prediction for \"validation dataset\" for each model's single trajectory prediction and optimize these parameters to get best score on this validation dataset.", "1001005": "Maybe we can select the three with the highest confidence from all the trajectories of different models, and then normalize their confidence.", "1001026": "I think it does not work well to include \"minor\" trajectory.\n\nSelected Top-3 trajectory may be a collapsed trajectory that goes same direction.", "1001032": "You're right.", "1001040": "corochann I saw your recent commit with muti pred confidence just now. That looks good, a 3 * (X,Y,Confs) output.", "1001600": "Thank you for checking update. Yes I surprised with the scores. I haven't done any parameter tuning yet and that result was from a single model :)", "1004657": "After training exactly using your notebook( same parameters) for 20000 iterations on kaggle lift dataset, My Lb score is 191 instead of 49. Is there something missing from your training notebook? or are you training on full train dataset available on lyft website instead of kaggle one?", "1004661": "Did you set `debug=False` to use `train.zarr` instead of `sample.zarr`?\nMy current score comes from using `train.zarr` for about 2 epochs I guess.", "1053172": "is there any update on ensembling of multi trajectory predictions?", "1053349": "corochann But 3 confidence ( sum of all 3 = 1 ).. we are prediction 3 trajectory of next 50 steps ?\nx001/y001 to x050/y050 -- 1st trajectory\nx101/y101 to x150/y150 -- 2nd trajectory\nx201/y201 to x250/y250 -- 3rd trajectory\n\nNeed to understand what are these x250/y250 predictions ?", "1053358": "Yes we are predicting 3 trajectory of next 50 steps.", "1053362": "But what are values for x051/y051 to x100/y100 and x151/y151 to x200/y200 ? not able understand what are these values represent ? @corochann", "1053363": "Does it exist in the submission.csv file? @seshurajup", "1053419": "Now i am clear. Thanks @corochann i miss understood with broken sequence", "1053473": "I noticed same discussion happed here: https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/177408#986337", "1053717": "corochann Did you train on the entire train.zarr (22M samples) or a subset of the kaggle training data?", "1053724": "I'm trying both cases.", "1055430": "corochann Hey. Trying to reach you guys (the whole team) via email. Could you check your letters from kaggle?" }, "source": "meta" }