Selecting the final two submissions was a bit tricky job. Even though the public test constituted with 50% of the Test Dataset but still the private dataset was unknown.
\nThanks to @pestipeti for his discussion link
\nSo now we know that the top 50% dataset was the public and the last 50% was private.
\nWhat I did was plotted the tsne of the two sets. And yes their distribution was almost similar.This gave me a bit confidence that there wont be much shake-up. Hence I chose the one which scored best on the public set.
\nHow did you all decided. Eager to hear from you.
\n", "messages": [ { "id": "1093120", "postDate": "11/27/2020 13:06:52", "content": "Selecting the final two submissions was a bit tricky job. Even though the public test constituted with 50% of the Test Dataset but still the private dataset was unknown.
\nThanks to @pestipeti for his discussion link
\nSo now we know that the top 50% dataset was the public and the last 50% was private.
\nWhat I did was plotted the tsne of the two sets. And yes their distribution was almost similar.This gave me a bit confidence that there wont be much shake-up. Hence I chose the one which scored best on the public set.
\nHow did you all decided. Eager to hear from you.
\n", "rawMarkdown": "Selecting the final two submissions was a bit tricky job. Even though the public test constituted with 50% of the Test Dataset but still the private dataset was unknown.\n\nThanks to @pestipeti for his discussion [link](https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/196796)\n\nSo now we know that the top 50% dataset was the public and the last 50% was private.\n\nWhat I did was plotted the tsne of the two sets. And yes their distribution was almost similar.This gave me a bit confidence that there wont be much shake-up. Hence I chose the one which scored best on the public set.\n\nHow did you all decided. Eager to hear from you.\n\n[Click here to see the plot](https://github.com/MiHarsh/Public_stuffs/blob/master/Testing_tsne-min.png)", "votes": null }, { "id": "1093129", "postDate": "11/27/2020 13:17:10", "content": "For us the internal validations using chopped validation set and the public leaderboard were quite consistent. We retrained our models in the last two weeks and were able to improve a tiny bit even on the final day.
\nWe selected our best public LB submission using 3 models and another very similar submission using only two models.
\nThe best public score also had the best private score for us.
Yes , the chopped validation set was just like a pre-score of the lb-score.
", "rawMarkdown": "Yes , the chopped validation set was just like a pre-score of the lb-score.", "votes": null }, { "id": "1093286", "postDate": "11/27/2020 15:39:49", "content": "I want to add, that we estimated that the single model also get 1st place by some margin. Otherwise we probably would have selected another ensemble as the 2nd submission.
", "rawMarkdown": "I want to add, that we estimated that the single model also get 1st place by some margin. Otherwise we probably would have selected another ensemble as the 2nd submission.", "votes": null } ], "comments": [ { "id": 1093129, "author_name": "gaborfodor", "author_url": "", "post_date": "11/27/2020 13:17:10", "content": "For us the internal validations using chopped validation set and the public leaderboard were quite consistent. We retrained our models in the last two weeks and were able to improve a tiny bit even on the final day.
\nWe selected our best public LB submission using 3 models and another very similar submission using only two models.
\nThe best public score also had the best private score for us.
Yes , the chopped validation set was just like a pre-score of the lb-score.
", "votes": null, "replies": [] } ] }, { "id": 1093286, "author_name": "ilu000", "author_url": "", "post_date": "11/27/2020 15:39:49", "content": "I want to add, that we estimated that the single model also get 1st place by some margin. Otherwise we probably would have selected another ensemble as the 2nd submission.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1093120": "Selecting the final two submissions was a bit tricky job. Even though the public test constituted with 50% of the Test Dataset but still the private dataset was unknown.\n\nThanks to @pestipeti for his discussion [link](https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/196796)\n\nSo now we know that the top 50% dataset was the public and the last 50% was private.\n\nWhat I did was plotted the tsne of the two sets. And yes their distribution was almost similar.This gave me a bit confidence that there wont be much shake-up. Hence I chose the one which scored best on the public set.\n\nHow did you all decided. Eager to hear from you.\n\n[Click here to see the plot](https://github.com/MiHarsh/Public_stuffs/blob/master/Testing_tsne-min.png)", "1093129": "For us the internal validations using chopped validation set and the public leaderboard were quite consistent. We retrained our models in the last two weeks and were able to improve a tiny bit even on the final day.\n\nWe selected our best public LB submission using 3 models and another very similar submission using only two models.\nThe best public score also had the best private score for us.", "1093136": "Yes , the chopped validation set was just like a pre-score of the lb-score.", "1093286": "1. best ensemble (it had both, best local validation score and best public LB score)\n2. best single model in case anything strange happend with one of the models in the ensemble", "1093926": "I want to add, that we estimated that the single model also get 1st place by some margin. Otherwise we probably would have selected another ensemble as the 2nd submission." }, "source": "meta" }