Dear Kaggle staff,
\nI want to ask you if it is possible to publish test data set when competition is ended.
\nI guess most of the participants (including me) have been struggled to find out why infer with high resolution of input images produces high score on LB.
\nIf possible, I want to investigate it after the competition.
\nThanks.
", "messages": [ { "id": "1688172", "postDate": "02/13/2022 12:43:47", "content": "\nDear Kaggle staff,
\nI want to ask you if it is possible to publish test data set when competition is ended.
\nI guess most of the participants (including me) have been struggled to find out why infer with high resolution of input images produces high score on LB.
\nIf possible, I want to investigate it after the competition.
\nThanks.
", "rawMarkdown": "addisonhoward \n@sohier \n\nDear Kaggle staff,\n\nI want to ask you if it is possible to publish test data set when competition is ended.\n\nI guess most of the participants (including me) have been struggled to find out why infer with high resolution of input images produces high score on LB.\n\nIf possible, I want to investigate it after the competition.\n\nThanks.", "votes": null }, { "id": "1690092", "postDate": "02/14/2022 17:25:07", "content": "The Competition hosts ultimately have the final say if they'd like to release the final test data set, but we typically prefer it not be released - primarily to allow participants to continue to make late submissions to the competition without the ground truth being fully available to impact models that are being tested against an unseen test set.
\nThis is the preferable situation for us, but not a hard requirement we hold - however the hosts will need to make that decision.
", "rawMarkdown": "The Competition hosts ultimately have the final say if they'd like to release the final test data set, but we typically prefer it not be released - primarily to allow participants to continue to make late submissions to the competition without the ground truth being fully available to impact models that are being tested against an unseen test set.\n\nThis is the *preferable* situation for us, but not a hard requirement we hold - however the hosts will need to make that decision.", "votes": null }, { "id": "1690695", "postDate": "02/15/2022 04:11:05", "content": "Hi @tatamikenn , I'm part of CSIRO's host team. Currently, the most likely case going forward is that we will keep the private test set private for the reason Addison mentioned, but we do plan to release more annotated videos/datasets which can be used as unseen test sets.
", "rawMarkdown": "Hi @tatamikenn , I'm part of CSIRO's host team. Currently, the most likely case going forward is that we will keep the private test set private for the reason Addison mentioned, but we do plan to release more annotated videos/datasets which can be used as unseen test sets.", "votes": null }, { "id": "1691208", "postDate": "02/15/2022 09:47:42", "content": "Wonderful. It’ll definitely help Kaggle community if it is released.
\nThank you for the reply.
By the way, I don’t know if CSIRO team noticed, we (some participants in the competition including me) found a lot of missing labels in the train dataset.
\nIf you are planning to release another dataset, I would recommend to label aided with more accurate model (e.g. this competition’s winner models) than the model you used to create the train labels. It will provide further accurate models if they are trained with it.
I somewhat understand the reasoning, but as to this competition dataset, I suspect the labels are not accurate enough. I, as well as some participants, noticed a lot of missing labels in the train dataset.
\nIf it is also true for the test labels, training to fit unseen inaccurate labels would only produce suboptimal experiment results. So I think it is more valuable to disclose entire dataset, enabling to check if the models prediction is truly wrong or not.
However, this is the matter the host decides.
", "rawMarkdown": "I somewhat understand the reasoning, but as to this competition dataset, I suspect the labels are not accurate enough. I, as well as some participants, noticed a lot of missing labels in the train dataset.\nIf it is also true for the test labels, training to fit unseen inaccurate labels would only produce suboptimal experiment results. So I think it is more valuable to disclose entire dataset, enabling to check if the models prediction is truly wrong or not.\n\nHowever, this is the matter the host decides.", "votes": null } ], "comments": [ { "id": 1690092, "author_name": "addisonhoward", "author_url": "", "post_date": "02/14/2022 17:25:07", "content": "The Competition hosts ultimately have the final say if they'd like to release the final test data set, but we typically prefer it not be released - primarily to allow participants to continue to make late submissions to the competition without the ground truth being fully available to impact models that are being tested against an unseen test set.
\nThis is the preferable situation for us, but not a hard requirement we hold - however the hosts will need to make that decision.
", "votes": null, "replies": [ { "id": 1691249, "author_name": "tatamikenn", "author_url": "", "post_date": "02/15/2022 10:06:41", "content": "I somewhat understand the reasoning, but as to this competition dataset, I suspect the labels are not accurate enough. I, as well as some participants, noticed a lot of missing labels in the train dataset.
\nIf it is also true for the test labels, training to fit unseen inaccurate labels would only produce suboptimal experiment results. So I think it is more valuable to disclose entire dataset, enabling to check if the models prediction is truly wrong or not.
However, this is the matter the host decides.
", "votes": null, "replies": [] } ] }, { "id": 1690695, "author_name": "ryanliu2021", "author_url": "", "post_date": "02/15/2022 04:11:05", "content": "Hi @tatamikenn , I'm part of CSIRO's host team. Currently, the most likely case going forward is that we will keep the private test set private for the reason Addison mentioned, but we do plan to release more annotated videos/datasets which can be used as unseen test sets.
", "votes": null, "replies": [ { "id": 1691208, "author_name": "tatamikenn", "author_url": "", "post_date": "02/15/2022 09:47:42", "content": "Wonderful. It’ll definitely help Kaggle community if it is released.
\nThank you for the reply.
By the way, I don’t know if CSIRO team noticed, we (some participants in the competition including me) found a lot of missing labels in the train dataset.
\nIf you are planning to release another dataset, I would recommend to label aided with more accurate model (e.g. this competition’s winner models) than the model you used to create the train labels. It will provide further accurate models if they are trained with it.