Ok, so here is a less-than-honest imho approach to the special task: Suppose one has developed a good algorithm (better than what can be achieved in a raspberry with a 200ms limit) , say one that requires a high end machine to run for a day. Then he could run his algorithm on said machine and predict the test set. Then he could train his raspberry algorithm using the results. The resulting algorithm is small, and over-trained on the test set, so it performs good on it, but not so good for any other set… (If we take this to the extreme, the raspberry part could be a checksum, and a lookup). It might be obvious, but I was wondering how it would be possible to protect against this.
", "messages": [ { "id": "260468", "postDate": "12/20/2017 08:54:33", "content": "Ok, so here is a less-than-honest imho approach to the special task: Suppose one has developed a good algorithm (better than what can be achieved in a raspberry with a 200ms limit) , say one that requires a high end machine to run for a day. Then he could run his algorithm on said machine and predict the test set. Then he could train his raspberry algorithm using the results. The resulting algorithm is small, and over-trained on the test set, so it performs good on it, but not so good for any other set… (If we take this to the extreme, the raspberry part could be a checksum, and a lookup). It might be obvious, but I was wondering how it would be possible to protect against this.
", "rawMarkdown": "Ok, so here is a less-than-honest imho approach to the special task: Suppose one has developed a good algorithm (better than what can be achieved in a raspberry with a 200ms limit) , say one that requires a high end machine to run for a day. Then he could run his algorithm on said machine and predict the test set. Then he could train his raspberry algorithm using the results. The resulting algorithm is small, and over-trained on the test set, so it performs good on it, but not so good for any other set… (If we take this to the extreme, the raspberry part could be a checksum, and a lookup). It might be obvious, but I was wondering how it would be possible to protect against this.", "votes": null }, { "id": "260505", "postDate": "12/20/2017 10:46:19", "content": "The public leaderboard is calculated with only 30% of the final test data. If you overfit to that youre gonna have a not so good result in the final standings.
", "rawMarkdown": "The public leaderboard is calculated with only 30% of the final test data. If you overfit to that youre gonna have a not so good result in the final standings.", "votes": null }, { "id": "260541", "postDate": "12/20/2017 12:23:39", "content": "There is an Honest way to do something similar : https://arxiv.org/pdf/1312.6184.pdf \nIts called model compression ( or Dark knowledge by Hinton)
", "rawMarkdown": "There is an Honest way to do something similar : https://arxiv.org/pdf/1312.6184.pdf \nIts called model compression ( or Dark knowledge by Hinton)", "votes": null }, { "id": "260605", "postDate": "12/20/2017 14:28:51", "content": "@Luigi: Right, but in this case, I am not over-fitting to the leaderboard's 30%, I am over-fitting to my prediction of the 100% of the test set, including the unknown 70%, where over-fitting is based not to the ground truth, but to my superior (and very time consuming) model. @CVxTz: Really cool paper & results! Indeed, this would definitely be Honest, which is probably why it seems to require more work and talent ;)
", "rawMarkdown": "Luigi: Right, but in this case, I am not over-fitting to the leaderboard's 30%, I am over-fitting to my prediction of the 100% of the test set, including the unknown 70%, where over-fitting is based not to the ground truth, but to my superior (and very time consuming) model. @CVxTz: Really cool paper & results! Indeed, this would definitely be Honest, which is probably why it seems to require more work and talent ;)", "votes": null }, { "id": "260624", "postDate": "12/20/2017 15:43:40", "content": "Biggest problem is that, if you win the prize, you need to submit a document and a reproduce guideline.
", "rawMarkdown": "Biggest problem is that, if you win the prize, you need to submit a document and a reproduce guideline.", "votes": null }, { "id": "260773", "postDate": "12/20/2017 21:58:22", "content": "I think what you described is called semi-supervised learning and/or pseudo labeling. A good idea to implement when it cost too much to acquire big data set that are hand labeled.\nGreat approach to take when it's too costly to acquire more data (like this competition since using outside data is prohibited) and/or your target deployment hardware is not powerful (like this competition if you are going for the special prize).\nI'm not sure why you think it is not honest... Make the good algorithm and claim the first prize money. Then, train a separate algorithm for the special prize and you can win up to $16K!!!\nI'm having hard time coming up with one algorithm that will place high in the leaderboard...
", "rawMarkdown": "I think what you described is called semi-supervised learning and/or pseudo labeling. A good idea to implement when it cost too much to acquire big data set that are hand labeled.\nGreat approach to take when it's too costly to acquire more data (like this competition since using outside data is prohibited) and/or your target deployment hardware is not powerful (like this competition if you are going for the special prize).\nI'm not sure why you think it is not honest... Make the good algorithm and claim the first prize money. Then, train a separate algorithm for the special prize and you can win up to $16K!!!\nI'm having hard time coming up with one algorithm that will place high in the leaderboard...", "votes": null }, { "id": "260807", "postDate": "12/20/2017 23:46:13", "content": "Yes, I wad pretty tired when I wrote this comment and didn't grasp the idea.\nI believe the technique is called transfer learning, and it's perfectly viable.
", "rawMarkdown": "Yes, I wad pretty tired when I wrote this comment and didn't grasp the idea.\nI believe the technique is called transfer learning, and it's perfectly viable.", "votes": null } ], "comments": [ { "id": 260505, "author_name": "lugi777", "author_url": "", "post_date": "12/20/2017 10:46:19", "content": "The public leaderboard is calculated with only 30% of the final test data. If you overfit to that youre gonna have a not so good result in the final standings.
", "votes": null, "replies": [] }, { "id": 260541, "author_name": "CVxTz", "author_url": "", "post_date": "12/20/2017 12:23:39", "content": "There is an Honest way to do something similar : https://arxiv.org/pdf/1312.6184.pdf \nIts called model compression ( or Dark knowledge by Hinton)
", "votes": null, "replies": [] }, { "id": 260605, "author_name": "nickthegreek", "author_url": "", "post_date": "12/20/2017 14:28:51", "content": "@Luigi: Right, but in this case, I am not over-fitting to the leaderboard's 30%, I am over-fitting to my prediction of the 100% of the test set, including the unknown 70%, where over-fitting is based not to the ground truth, but to my superior (and very time consuming) model. @CVxTz: Really cool paper & results! Indeed, this would definitely be Honest, which is probably why it seems to require more work and talent ;)
", "votes": null, "replies": [ { "id": 260807, "author_name": "lugi777", "author_url": "", "post_date": "12/20/2017 23:46:13", "content": "Yes, I wad pretty tired when I wrote this comment and didn't grasp the idea.\nI believe the technique is called transfer learning, and it's perfectly viable.
", "votes": null, "replies": [] } ] }, { "id": 260624, "author_name": "ildoonet", "author_url": "", "post_date": "12/20/2017 15:43:40", "content": "Biggest problem is that, if you win the prize, you need to submit a document and a reproduce guideline.
", "votes": null, "replies": [] }, { "id": 260773, "author_name": "bsp2020", "author_url": "", "post_date": "12/20/2017 21:58:22", "content": "I think what you described is called semi-supervised learning and/or pseudo labeling. A good idea to implement when it cost too much to acquire big data set that are hand labeled.\nGreat approach to take when it's too costly to acquire more data (like this competition since using outside data is prohibited) and/or your target deployment hardware is not powerful (like this competition if you are going for the special prize).\nI'm not sure why you think it is not honest... Make the good algorithm and claim the first prize money. Then, train a separate algorithm for the special prize and you can win up to $16K!!!\nI'm having hard time coming up with one algorithm that will place high in the leaderboard...
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "260468": "Ok, so here is a less-than-honest imho approach to the special task: Suppose one has developed a good algorithm (better than what can be achieved in a raspberry with a 200ms limit) , say one that requires a high end machine to run for a day. Then he could run his algorithm on said machine and predict the test set. Then he could train his raspberry algorithm using the results. The resulting algorithm is small, and over-trained on the test set, so it performs good on it, but not so good for any other set… (If we take this to the extreme, the raspberry part could be a checksum, and a lookup). It might be obvious, but I was wondering how it would be possible to protect against this.", "260505": "The public leaderboard is calculated with only 30% of the final test data. If you overfit to that youre gonna have a not so good result in the final standings.", "260541": "There is an Honest way to do something similar : https://arxiv.org/pdf/1312.6184.pdf \nIts called model compression ( or Dark knowledge by Hinton)", "260605": "Luigi: Right, but in this case, I am not over-fitting to the leaderboard's 30%, I am over-fitting to my prediction of the 100% of the test set, including the unknown 70%, where over-fitting is based not to the ground truth, but to my superior (and very time consuming) model. @CVxTz: Really cool paper & results! Indeed, this would definitely be Honest, which is probably why it seems to require more work and talent ;)", "260624": "Biggest problem is that, if you win the prize, you need to submit a document and a reproduce guideline.", "260773": "I think what you described is called semi-supervised learning and/or pseudo labeling. A good idea to implement when it cost too much to acquire big data set that are hand labeled.\nGreat approach to take when it's too costly to acquire more data (like this competition since using outside data is prohibited) and/or your target deployment hardware is not powerful (like this competition if you are going for the special prize).\nI'm not sure why you think it is not honest... Make the good algorithm and claim the first prize money. Then, train a separate algorithm for the special prize and you can win up to $16K!!!\nI'm having hard time coming up with one algorithm that will place high in the leaderboard...", "260807": "Yes, I wad pretty tired when I wrote this comment and didn't grasp the idea.\nI believe the technique is called transfer learning, and it's perfectly viable." }, "source": "meta" }