This challenge seems like an obvious case for unsupervised pretraining, and training on the test data would be useful.
\n\nSo, here is the question, both to the other competitors, but also to the administrators, would it be frowned upon or worse disqualifying to use the test data for unsupervised learning. If not disqualifying can anyone give a good logical reason why I would not want to do that?
", "messages": [ { "id": "107217", "postDate": "02/08/2016 13:45:07", "content": "This challenge seems like an obvious case for unsupervised pretraining, and training on the test data would be useful.
\n\nSo, here is the question, both to the other competitors, but also to the administrators, would it be frowned upon or worse disqualifying to use the test data for unsupervised learning. If not disqualifying can anyone give a good logical reason why I would not want to do that?
", "rawMarkdown": "This challenge seems like an obvious case for unsupervised pretraining, and training on the test data would be useful. \r\n\r\nSo, here is the question, both to the other competitors, but also to the administrators, would it be frowned upon or worse disqualifying to use the test data for unsupervised learning. If not disqualifying can anyone give a good logical reason why I would not want to do that?", "votes": null }, { "id": "107451", "postDate": "02/09/2016 23:30:43", "content": "For this contest we do not have any rule that prohibits using unsupervised learning that includes test data, so yes you are allowed to do that as part of your prediction pipeline.
\n\nHowever in practice for ML cross-validation, you should avoid using any of your test data in building your model. Otherwise your model 'knows' something it shouldn't know, and the cross-validation metrics might not be a good estimate of the model's performance on unseen data in future prediction tasks.
", "rawMarkdown": "For this contest we do not have any rule that prohibits using unsupervised learning that includes test data, so yes you are allowed to do that as part of your prediction pipeline.\r\n\r\nHowever in practice for ML cross-validation, you should avoid using any of your test data in building your model. Otherwise your model 'knows' something it shouldn't know, and the cross-validation metrics might not be a good estimate of the model's performance on unseen data in future prediction tasks.", "votes": null }, { "id": "107503", "postDate": "02/10/2016 13:34:15", "content": "Yes, I will definitely not do unsupervised learning on my own hold out validation and and test data, but only the photos in the competitions Test data, and I do suspect that this practice still will cause the second 70% of the test data that I will never see until the end to perform significantly worse than than my leader board results.
", "rawMarkdown": "Yes, I will definitely not do unsupervised learning on my own hold out validation and and test data, but only the photos in the competitions Test data, and I do suspect that this practice still will cause the second 70% of the test data that I will never see until the end to perform significantly worse than than my leader board results.", "votes": null } ], "comments": [ { "id": 107451, "author_name": "fcchou", "author_url": "", "post_date": "02/09/2016 23:30:43", "content": "For this contest we do not have any rule that prohibits using unsupervised learning that includes test data, so yes you are allowed to do that as part of your prediction pipeline.
\n\nHowever in practice for ML cross-validation, you should avoid using any of your test data in building your model. Otherwise your model 'knows' something it shouldn't know, and the cross-validation metrics might not be a good estimate of the model's performance on unseen data in future prediction tasks.
", "votes": null, "replies": [] }, { "id": 107503, "author_name": "joexjmmvhm", "author_url": "", "post_date": "02/10/2016 13:34:15", "content": "Yes, I will definitely not do unsupervised learning on my own hold out validation and and test data, but only the photos in the competitions Test data, and I do suspect that this practice still will cause the second 70% of the test data that I will never see until the end to perform significantly worse than than my leader board results.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "107217": "This challenge seems like an obvious case for unsupervised pretraining, and training on the test data would be useful. \r\n\r\nSo, here is the question, both to the other competitors, but also to the administrators, would it be frowned upon or worse disqualifying to use the test data for unsupervised learning. If not disqualifying can anyone give a good logical reason why I would not want to do that?", "107451": "For this contest we do not have any rule that prohibits using unsupervised learning that includes test data, so yes you are allowed to do that as part of your prediction pipeline.\r\n\r\nHowever in practice for ML cross-validation, you should avoid using any of your test data in building your model. Otherwise your model 'knows' something it shouldn't know, and the cross-validation metrics might not be a good estimate of the model's performance on unseen data in future prediction tasks.", "107503": "Yes, I will definitely not do unsupervised learning on my own hold out validation and and test data, but only the photos in the competitions Test data, and I do suspect that this practice still will cause the second 70% of the test data that I will never see until the end to perform significantly worse than than my leader board results." }, "source": "meta" }