It is obvious that people who have a hand labelled dataset will now keep the true labels hidden to avoid any prying eyes , these people will ruin the private leaderboard as well , you may think it's not possible but just assume if someone has a true labels for 87 data points (20%) , they can simply use it as their validation set for training , which means their model will have more information to train and learn from , I don't even kaggle will be able to pin point such users .
\nAlso a note to everyone who is looking for hand labelled , there are instances where people share such information on various social media messaging platforms which are end to end encrypted .
\nWhat's your take on this ?
", "messages": [ { "id": "1502388", "postDate": "09/04/2021 07:46:22", "content": "It is obvious that people who have a hand labelled dataset will now keep the true labels hidden to avoid any prying eyes , these people will ruin the private leaderboard as well , you may think it's not possible but just assume if someone has a true labels for 87 data points (20%) , they can simply use it as their validation set for training , which means their model will have more information to train and learn from , I don't even kaggle will be able to pin point such users .
\nAlso a note to everyone who is looking for hand labelled , there are instances where people share such information on various social media messaging platforms which are end to end encrypted .
\nWhat's your take on this ?
", "rawMarkdown": "It is obvious that people who have a hand labelled dataset will now keep the true labels hidden to avoid any prying eyes , these people will ruin the private leaderboard as well , you may think it's not possible but just assume if someone has a true labels for 87 data points (20%) , they can simply use it as their validation set for training , which means their model will have more information to train and learn from , I don't even kaggle will be able to pin point such users . \n\nAlso a note to everyone who is looking for hand labelled , there are instances where people share such information on various social media messaging platforms which are end to end encrypted . \n\n\nWhat's your take on this ?", "votes": null }, { "id": "1502656", "postDate": "09/04/2021 14:04:45", "content": "I agree. I think access to labelled public test dataset gives unfair advantage to teams who have them but LB probing is very hard to prevent. To be fair, I hope the host can release the groundtruth labels near the end of the competition to allow everyone to have a chance to use the public LB test set as additional training data.
\nI remembered that past RSNA competitions had 2 stages. In stage 2, the labels for stage 1 test are disclosed to allow people to retrain their models if they so choose. It also negates the advantage for teams who hand-labelled. Maybe this competition can take a similar form.
", "rawMarkdown": "I agree. I think access to labelled public test dataset gives unfair advantage to teams who have them but LB probing is very hard to prevent. To be fair, I hope the host can release the groundtruth labels near the end of the competition to allow everyone to have a chance to use the public LB test set as additional training data. \n\nI remembered that past RSNA competitions had 2 stages. In stage 2, the labels for stage 1 test are disclosed to allow people to retrain their models if they so choose. It also negates the advantage for teams who hand-labelled. Maybe this competition can take a similar form.", "votes": null }, { "id": "1502678", "postDate": "09/04/2021 14:32:59", "content": "Let's see what the organizers have planned for this .
", "rawMarkdown": "Let's see what the organizers have planned for this .", "votes": null }, { "id": "1502700", "postDate": "09/04/2021 14:52:35", "content": "@yeeseng @avikrams I agree with you about releasing of public test data ground label. However, there is no point in releasing them at the end of the competition, teams who've already done hand labeling and have access to public test data labels would have a big advantage of training their models and tuning parameters by validating with test data.
\nI think releasing at the end of the competition would not fix things completely, it's better to release them at the earlierst.
\nAlso, I would appreciate if hosts have a better plan for it.
@nischaydnk i have not participated in any competitions where the data labels for test was released , I think @yeeseng can give you better views in this regards
", "rawMarkdown": "nischaydnk i have not participated in any competitions where the data labels for test was released , I think @yeeseng can give you better views in this regards", "votes": null }, { "id": "1503762", "postDate": "09/05/2021 17:23:02", "content": "Here's an FAQ about 2-stage competitions:
\nhttps://www.kaggle.com/two-stage-frequently-asked-questions
An example of a 2-stage competition, click on 'timeline' of the competition:
\nhttps://www.kaggle.com/c/rsna-intracranial-hemorrhage-detection
It seems like RSNA has moved away from 2-stage competitions because for most part it is not necessary. It is difficult to hand label thousands of radiology exams in the public LB set of past competitions. But in this case where the test set contained only 87 studies, it just might made sense to hand-label or LB probe each case.
\n2 weeks are not a lot of time but sufficient to retrain any already optimized models. Any longer would further reduce the usefulness of the public LB as a gauge of your standing.
", "rawMarkdown": "Here's an FAQ about 2-stage competitions: \nhttps://www.kaggle.com/two-stage-frequently-asked-questions\n\nAn example of a 2-stage competition, click on 'timeline' of the competition:\nhttps://www.kaggle.com/c/rsna-intracranial-hemorrhage-detection\n\nIt seems like RSNA has moved away from 2-stage competitions because for most part it is not necessary. It is difficult to hand label thousands of radiology exams in the public LB set of past competitions. But in this case where the test set contained only 87 studies, it just might made sense to hand-label or LB probe each case.\n\n2 weeks are not a lot of time but sufficient to retrain any already optimized models. Any longer would further reduce the usefulness of the public LB as a gauge of your standing.", "votes": null }, { "id": "1504022", "postDate": "09/06/2021 02:30:25", "content": "Having labels will help in optimising for the public leaderboard, not sure if these optimisations will generalise to the private lb dataset.
\nWhat it really helps with is not having to make a train/test split on the already scarce training data which is pretty helpful tbh.
Also the rules say you cant train on HL test data, not sure if that is checked in the final submission.
", "rawMarkdown": "Having labels will help in optimising for the public leaderboard, not sure if these optimisations will generalise to the private lb dataset. \nWhat it really helps with is not having to make a train/test split on the already scarce training data which is pretty helpful tbh.\n\nAlso the rules say you cant train on HL test data, not sure if that is checked in the final submission.", "votes": null } ], "comments": [ { "id": 1502656, "author_name": "yeeseng", "author_url": "", "post_date": "09/04/2021 14:04:45", "content": "I agree. I think access to labelled public test dataset gives unfair advantage to teams who have them but LB probing is very hard to prevent. To be fair, I hope the host can release the groundtruth labels near the end of the competition to allow everyone to have a chance to use the public LB test set as additional training data.
\nI remembered that past RSNA competitions had 2 stages. In stage 2, the labels for stage 1 test are disclosed to allow people to retrain their models if they so choose. It also negates the advantage for teams who hand-labelled. Maybe this competition can take a similar form.
", "votes": null, "replies": [ { "id": 1502678, "author_name": "avikrams", "author_url": "", "post_date": "09/04/2021 14:32:59", "content": "Let's see what the organizers have planned for this .
", "votes": null, "replies": [] }, { "id": 1502700, "author_name": "nischaydnk", "author_url": "", "post_date": "09/04/2021 14:52:35", "content": "@yeeseng @avikrams I agree with you about releasing of public test data ground label. However, there is no point in releasing them at the end of the competition, teams who've already done hand labeling and have access to public test data labels would have a big advantage of training their models and tuning parameters by validating with test data.
\nI think releasing at the end of the competition would not fix things completely, it's better to release them at the earlierst.
\nAlso, I would appreciate if hosts have a better plan for it.
@nischaydnk i have not participated in any competitions where the data labels for test was released , I think @yeeseng can give you better views in this regards
", "votes": null, "replies": [] }, { "id": 1503762, "author_name": "yeeseng", "author_url": "", "post_date": "09/05/2021 17:23:02", "content": "Here's an FAQ about 2-stage competitions:
\nhttps://www.kaggle.com/two-stage-frequently-asked-questions
An example of a 2-stage competition, click on 'timeline' of the competition:
\nhttps://www.kaggle.com/c/rsna-intracranial-hemorrhage-detection
It seems like RSNA has moved away from 2-stage competitions because for most part it is not necessary. It is difficult to hand label thousands of radiology exams in the public LB set of past competitions. But in this case where the test set contained only 87 studies, it just might made sense to hand-label or LB probe each case.
\n2 weeks are not a lot of time but sufficient to retrain any already optimized models. Any longer would further reduce the usefulness of the public LB as a gauge of your standing.
", "votes": null, "replies": [] } ] }, { "id": 1504022, "author_name": "aryamansharma47", "author_url": "", "post_date": "09/06/2021 02:30:25", "content": "Having labels will help in optimising for the public leaderboard, not sure if these optimisations will generalise to the private lb dataset.
\nWhat it really helps with is not having to make a train/test split on the already scarce training data which is pretty helpful tbh.
Also the rules say you cant train on HL test data, not sure if that is checked in the final submission.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1502388": "It is obvious that people who have a hand labelled dataset will now keep the true labels hidden to avoid any prying eyes , these people will ruin the private leaderboard as well , you may think it's not possible but just assume if someone has a true labels for 87 data points (20%) , they can simply use it as their validation set for training , which means their model will have more information to train and learn from , I don't even kaggle will be able to pin point such users . \n\nAlso a note to everyone who is looking for hand labelled , there are instances where people share such information on various social media messaging platforms which are end to end encrypted . \n\n\nWhat's your take on this ?", "1502656": "I agree. I think access to labelled public test dataset gives unfair advantage to teams who have them but LB probing is very hard to prevent. To be fair, I hope the host can release the groundtruth labels near the end of the competition to allow everyone to have a chance to use the public LB test set as additional training data. \n\nI remembered that past RSNA competitions had 2 stages. In stage 2, the labels for stage 1 test are disclosed to allow people to retrain their models if they so choose. It also negates the advantage for teams who hand-labelled. Maybe this competition can take a similar form.", "1502678": "Let's see what the organizers have planned for this .", "1502700": "yeeseng @avikrams I agree with you about releasing of public test data ground label. However, there is no point in releasing them at the end of the competition, teams who've already done hand labeling and have access to public test data labels would have a big advantage of training their models and tuning parameters by validating with test data. \nI think releasing at the end of the competition would not fix things completely, it's better to release them at the earlierst. \nAlso, I would appreciate if hosts have a better plan for it.", "1502976": "nischaydnk i have not participated in any competitions where the data labels for test was released , I think @yeeseng can give you better views in this regards", "1503762": "Here's an FAQ about 2-stage competitions: \nhttps://www.kaggle.com/two-stage-frequently-asked-questions\n\nAn example of a 2-stage competition, click on 'timeline' of the competition:\nhttps://www.kaggle.com/c/rsna-intracranial-hemorrhage-detection\n\nIt seems like RSNA has moved away from 2-stage competitions because for most part it is not necessary. It is difficult to hand label thousands of radiology exams in the public LB set of past competitions. But in this case where the test set contained only 87 studies, it just might made sense to hand-label or LB probe each case.\n\n2 weeks are not a lot of time but sufficient to retrain any already optimized models. Any longer would further reduce the usefulness of the public LB as a gauge of your standing.", "1504022": "Having labels will help in optimising for the public leaderboard, not sure if these optimisations will generalise to the private lb dataset. \nWhat it really helps with is not having to make a train/test split on the already scarce training data which is pretty helpful tbh.\n\nAlso the rules say you cant train on HL test data, not sure if that is checked in the final submission." }, "source": "meta" }