I'm trying really hard to not get sucked into another Kaggle competition, because Kaggle is addictive, but I've already done three competitions in a row and I need to take a break. So one way to motivate myself not to compete here is to just share my idea so I can't benefit from it. (Sorry but not sorry if other people were hoarding this idea to themselves and I blew your cover. Also sorry but not sorry if this idea is bad and makes your models even more overfit and you lose.)
\n\nBasically, I see a good deal here that looks like the Jigsaw toxic Kaggle competition that just finished -- really high AUCs and questionable train / test set differences. I was really surprised there to to see the first place winner winner do so well by using their best ensemble to label the test set data, predict those labels, and then repeat until convergence. Thinking about it, it definitely makes sense as a strategy, especially when your AUCs are high. Given that we also have an extra large supplemental test set, this could be really useful.
", "messages": [ { "id": "301656", "postDate": "03/23/2018 03:45:03", "content": "I'm trying really hard to not get sucked into another Kaggle competition, because Kaggle is addictive, but I've already done three competitions in a row and I need to take a break. So one way to motivate myself not to compete here is to just share my idea so I can't benefit from it. (Sorry but not sorry if other people were hoarding this idea to themselves and I blew your cover. Also sorry but not sorry if this idea is bad and makes your models even more overfit and you lose.)
\n\nBasically, I see a good deal here that looks like the Jigsaw toxic Kaggle competition that just finished -- really high AUCs and questionable train / test set differences. I was really surprised there to to see the first place winner winner do so well by using their best ensemble to label the test set data, predict those labels, and then repeat until convergence. Thinking about it, it definitely makes sense as a strategy, especially when your AUCs are high. Given that we also have an extra large supplemental test set, this could be really useful.
", "rawMarkdown": "I'm trying really hard to not get sucked into another Kaggle competition, because Kaggle is addictive, but I've already done three competitions in a row and I need to take a break. So one way to motivate myself not to compete here is to just share my idea so I can't benefit from it. (Sorry but not sorry if other people were hoarding this idea to themselves and I blew your cover. Also sorry but not sorry if this idea is bad and makes your models even more overfit and you lose.)\n\nBasically, I see a good deal here that looks like the Jigsaw toxic Kaggle competition that just finished -- really high AUCs and questionable train / test set differences. I was really surprised there to to see the [first place winner](https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/discussion/52557) winner do so well by using their best ensemble to label the test set data, predict those labels, and then repeat until convergence. Thinking about it, it definitely makes sense as a strategy, especially when your AUCs are high. Given that we also have an extra large supplemental test set, this could be really useful.", "votes": null }, { "id": "301689", "postDate": "03/23/2018 05:41:24", "content": "Put in to do list. Thanks!
\n\nSo according to them this is make sense when there is a difference in train and test distributions?
", "rawMarkdown": "Put in to do list. Thanks!\n\nSo according to them this is make sense when there is a difference in train and test distributions?", "votes": null }, { "id": "302105", "postDate": "03/23/2018 17:39:31", "content": "It seems theoretically to make sense whenever you can guess test labels with really high accuracy.
", "rawMarkdown": "It seems theoretically to make sense whenever you can guess test labels with really high accuracy.", "votes": null }, { "id": "302179", "postDate": "03/23/2018 19:12:50", "content": "Peter, even if it's early to say I wouldn't bet on it;
\n\nPseudo-labeling is a trade off , its effect on one hand depends on accuracy , as you have said. But just as important, it depends on how much you gain when you add data to train.
\n\nIn toxic competition there was not so much data for the complexity of the task, many symptoms apart of the number of less frequent labels, pretrained embeddings worked better than custom, high number of out of voc... Good setup for pseudo -labeling.
\n\nI entered toxic really late so a ton of things I didn't try but pseudolabelling was definitely high in my list then. In this one its not even in the list. But I could be wrong...
\n\nAnyway... c'mon man, it's not too late, why not join the party? Its not such a bad adiction! :)
", "rawMarkdown": "Peter, even if it's early to say I wouldn't bet on it; \n\n**Pseudo-labeling is a trade off** , its effect on one hand depends on accuracy , as you have said. But just as important, it depends on how much you gain when you add data to train.\n\nIn toxic competition there was not so much data for the complexity of the task, many symptoms apart of the number of less frequent labels, pretrained embeddings worked better than custom, high number of out of voc... Good setup for pseudo -labeling. \n\nI entered toxic really late so a ton of things I didn't try but pseudolabelling was definitely high in my list then. In this one its not even in the list. But I could be wrong...\n\nAnyway... c'mon man, it's not too late, why not join the party? Its not such a bad adiction! :)", "votes": null }, { "id": "302190", "postDate": "03/23/2018 19:27:15", "content": "\n\n\nit's not too late, why not join the party? Its not such a bad adiction! :)
\n
I tell myself that a lot... there are much worse things to have an addiction to. :) But I've been making doing well on Kaggle pretty much my #1 priority for the past five months now. I'm very delighted with the results I've shown for it on Kaggle, but I need to pick up slack on the things I've been neglecting.
\n\nBut don't worry, I'll rejoin y'all soon enough! It's going to be a long road to Grandmaster. :)
", "rawMarkdown": "> it's not too late, why not join the party? Its not such a bad adiction! :)\n\nI tell myself that a lot... there are much worse things to have an addiction to. :) But I've been making doing well on Kaggle pretty much my #1 priority for the past five months now. I'm very delighted with the results I've shown for it on Kaggle, but I need to pick up slack on the things I've been neglecting.\n\nBut don't worry, I'll rejoin y'all soon enough! It's going to be a long road to Grandmaster. :)", "votes": null }, { "id": "302194", "postDate": "03/23/2018 19:28:58", "content": "> In toxic competition there was not so much data for the complexity of the task, many symptoms apart of the number of less frequent labels, pretrained embeddings worked better than custom, high number of out of voc... Good setup for pseudo -labeling.
\n\nThat's really interesting. I didn't get a sense here that the data size was that different from toxic, but the fact that this competition is not text-based is definitely a huge difference.
", "rawMarkdown": "> In toxic competition there was not so much data for the complexity of the task, many symptoms apart of the number of less frequent labels, pretrained embeddings worked better than custom, high number of out of voc... Good setup for pseudo -labeling.\n\nThat's really interesting. I didn't get a sense here that the data size was that different from toxic, but the fact that this competition is not text-based is definitely a huge difference.", "votes": null }, { "id": "320594", "postDate": "04/29/2018 07:10:43", "content": "I agree with miguel on this. The number of samples on here is massive in comparison to the toxic comment challenge. Seems like that extra bit of single you might be able to suss out would either be very small since you already have millions of samples to already learn from or simply incorrect because the current results aren't really all that good.
", "rawMarkdown": "I agree with miguel on this. The number of samples on here is massive in comparison to the toxic comment challenge. Seems like that extra bit of single you might be able to suss out would either be very small since you already have millions of samples to already learn from or simply incorrect because the current results aren't really all that good.", "votes": null }, { "id": "320649", "postDate": "04/29/2018 11:55:40", "content": "@peterhurford only a few days to go, so the damage is limited, dive into it and have some fun :p
", "rawMarkdown": "peterhurford only a few days to go, so the damage is limited, dive into it and have some fun :p", "votes": null }, { "id": "320723", "postDate": "04/29/2018 16:11:39", "content": "Thanks! I'm going to do the Avito Demand Competition though. :)
", "rawMarkdown": "Thanks! I'm going to do the [Avito Demand Competition](https://www.kaggle.com/c/avito-demand-prediction/) though. :)", "votes": null } ], "comments": [ { "id": 301689, "author_name": "muhammadalfiansyah", "author_url": "", "post_date": "03/23/2018 05:41:24", "content": "Put in to do list. Thanks!
\n\nSo according to them this is make sense when there is a difference in train and test distributions?
", "votes": null, "replies": [ { "id": 302105, "author_name": "peterhurford", "author_url": "", "post_date": "03/23/2018 17:39:31", "content": "It seems theoretically to make sense whenever you can guess test labels with really high accuracy.
", "votes": null, "replies": [] } ] }, { "id": 302179, "author_name": "miguelpm", "author_url": "", "post_date": "03/23/2018 19:12:50", "content": "Peter, even if it's early to say I wouldn't bet on it;
\n\nPseudo-labeling is a trade off , its effect on one hand depends on accuracy , as you have said. But just as important, it depends on how much you gain when you add data to train.
\n\nIn toxic competition there was not so much data for the complexity of the task, many symptoms apart of the number of less frequent labels, pretrained embeddings worked better than custom, high number of out of voc... Good setup for pseudo -labeling.
\n\nI entered toxic really late so a ton of things I didn't try but pseudolabelling was definitely high in my list then. In this one its not even in the list. But I could be wrong...
\n\nAnyway... c'mon man, it's not too late, why not join the party? Its not such a bad adiction! :)
", "votes": null, "replies": [ { "id": 302190, "author_name": "peterhurford", "author_url": "", "post_date": "03/23/2018 19:27:15", "content": "\n\n\nit's not too late, why not join the party? Its not such a bad adiction! :)
\n
I tell myself that a lot... there are much worse things to have an addiction to. :) But I've been making doing well on Kaggle pretty much my #1 priority for the past five months now. I'm very delighted with the results I've shown for it on Kaggle, but I need to pick up slack on the things I've been neglecting.
\n\nBut don't worry, I'll rejoin y'all soon enough! It's going to be a long road to Grandmaster. :)
", "votes": null, "replies": [] }, { "id": 302194, "author_name": "peterhurford", "author_url": "", "post_date": "03/23/2018 19:28:58", "content": "> In toxic competition there was not so much data for the complexity of the task, many symptoms apart of the number of less frequent labels, pretrained embeddings worked better than custom, high number of out of voc... Good setup for pseudo -labeling.
\n\nThat's really interesting. I didn't get a sense here that the data size was that different from toxic, but the fact that this competition is not text-based is definitely a huge difference.
", "votes": null, "replies": [] }, { "id": 320594, "author_name": "ryches", "author_url": "", "post_date": "04/29/2018 07:10:43", "content": "I agree with miguel on this. The number of samples on here is massive in comparison to the toxic comment challenge. Seems like that extra bit of single you might be able to suss out would either be very small since you already have millions of samples to already learn from or simply incorrect because the current results aren't really all that good.
", "votes": null, "replies": [] } ] }, { "id": 320649, "author_name": "yifanxie", "author_url": "", "post_date": "04/29/2018 11:55:40", "content": "@peterhurford only a few days to go, so the damage is limited, dive into it and have some fun :p
", "votes": null, "replies": [ { "id": 320723, "author_name": "peterhurford", "author_url": "", "post_date": "04/29/2018 16:11:39", "content": "Thanks! I'm going to do the Avito Demand Competition though. :)
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "301656": "I'm trying really hard to not get sucked into another Kaggle competition, because Kaggle is addictive, but I've already done three competitions in a row and I need to take a break. So one way to motivate myself not to compete here is to just share my idea so I can't benefit from it. (Sorry but not sorry if other people were hoarding this idea to themselves and I blew your cover. Also sorry but not sorry if this idea is bad and makes your models even more overfit and you lose.)\n\nBasically, I see a good deal here that looks like the Jigsaw toxic Kaggle competition that just finished -- really high AUCs and questionable train / test set differences. I was really surprised there to to see the [first place winner](https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/discussion/52557) winner do so well by using their best ensemble to label the test set data, predict those labels, and then repeat until convergence. Thinking about it, it definitely makes sense as a strategy, especially when your AUCs are high. Given that we also have an extra large supplemental test set, this could be really useful.", "301689": "Put in to do list. Thanks!\n\nSo according to them this is make sense when there is a difference in train and test distributions?", "302105": "It seems theoretically to make sense whenever you can guess test labels with really high accuracy.", "302179": "Peter, even if it's early to say I wouldn't bet on it; \n\n**Pseudo-labeling is a trade off** , its effect on one hand depends on accuracy , as you have said. But just as important, it depends on how much you gain when you add data to train.\n\nIn toxic competition there was not so much data for the complexity of the task, many symptoms apart of the number of less frequent labels, pretrained embeddings worked better than custom, high number of out of voc... Good setup for pseudo -labeling. \n\nI entered toxic really late so a ton of things I didn't try but pseudolabelling was definitely high in my list then. In this one its not even in the list. But I could be wrong...\n\nAnyway... c'mon man, it's not too late, why not join the party? Its not such a bad adiction! :)", "302190": "> it's not too late, why not join the party? Its not such a bad adiction! :)\n\nI tell myself that a lot... there are much worse things to have an addiction to. :) But I've been making doing well on Kaggle pretty much my #1 priority for the past five months now. I'm very delighted with the results I've shown for it on Kaggle, but I need to pick up slack on the things I've been neglecting.\n\nBut don't worry, I'll rejoin y'all soon enough! It's going to be a long road to Grandmaster. :)", "302194": "> In toxic competition there was not so much data for the complexity of the task, many symptoms apart of the number of less frequent labels, pretrained embeddings worked better than custom, high number of out of voc... Good setup for pseudo -labeling.\n\nThat's really interesting. I didn't get a sense here that the data size was that different from toxic, but the fact that this competition is not text-based is definitely a huge difference.", "320594": "I agree with miguel on this. The number of samples on here is massive in comparison to the toxic comment challenge. Seems like that extra bit of single you might be able to suss out would either be very small since you already have millions of samples to already learn from or simply incorrect because the current results aren't really all that good.", "320649": "peterhurford only a few days to go, so the damage is limited, dive into it and have some fun :p", "320723": "Thanks! I'm going to do the [Avito Demand Competition](https://www.kaggle.com/c/avito-demand-prediction/) though. :)" }, "source": "meta" }