Based on the Stanford OpenVaccine competition, it seems pseudo-labeling played a big part in many of the top solutions. However, since not many people are talking about it here, is it not likely to provide a big boost in this competition? Has anyone tried it?
", "messages": [ { "id": "2540860", "postDate": "11/28/2023 04:00:37", "content": "Based on the Stanford OpenVaccine competition, it seems pseudo-labeling played a big part in many of the top solutions. However, since not many people are talking about it here, is it not likely to provide a big boost in this competition? Has anyone tried it?
", "rawMarkdown": "Based on the [Stanford OpenVaccine competition](https://www.kaggle.com/competitions/stanford-covid-vaccine/overview), it seems pseudo-labeling played a big part in many of the top solutions. However, since not many people are talking about it here, is it not likely to provide a big boost in this competition? Has anyone tried it?", "votes": null }, { "id": "2541125", "postDate": "11/28/2023 08:50:26", "content": "From https://www.kaggle.com/code/cdeotte/pseudo-labeling-qda-0-969 \"Pseudo labeling is the process of adding confident predicted test data to your training data.\"
\nIn such case you can define a high confidence prediction as one closer enough to one at the final softmax layer. But how to define it in a regression problem like this one?
It depends in how good are you models at predicting, adding pseudo-labeling got me from 0.15092 to 0.14791 in single model, same architecture/hypers, took 6 times longer (2d 11h 57m 32s) to train though, that was using the pseudo-label + base data, need to finetune with the base data another one trained only with pseudo-labels to see if it makes any difference.
", "rawMarkdown": "It depends in how good are you models at predicting, adding pseudo-labeling got me from 0.15092 to 0.14791 in single model, same architecture/hypers, took 6 times longer (2d 11h 57m 32s) to train though, that was using the pseudo-label + base data, need to finetune with the base data another one trained only with pseudo-labels to see if it makes any difference.", "votes": null }, { "id": "2541190", "postDate": "11/28/2023 09:34:16", "content": "https://arxiv.org/pdf/2206.14486.pdf
\nNot specific to this dataset/problem, but good paper to think about the data.
", "rawMarkdown": "https://arxiv.org/pdf/2206.14486.pdf\n\nNot specific to this dataset/problem, but good paper to think about the data.", "votes": null }, { "id": "2541219", "postDate": "11/28/2023 09:55:13", "content": "Thanks, I'll check it.
", "rawMarkdown": "Thanks, I'll check it.", "votes": null }, { "id": "2541286", "postDate": "11/28/2023 11:08:06", "content": "Pseudo lables will only help once you generalize \"good enough\" for longer sequences. If you have bad generalization they will only add noise, which is why i don't think its been talked about.
", "rawMarkdown": "Pseudo lables will only help once you generalize \"good enough\" for longer sequences. If you have bad generalization they will only add noise, which is why i don't think its been talked about.", "votes": null } ], "comments": [ { "id": 2541125, "author_name": "sacuscreed", "author_url": "", "post_date": "11/28/2023 08:50:26", "content": "From https://www.kaggle.com/code/cdeotte/pseudo-labeling-qda-0-969 \"Pseudo labeling is the process of adding confident predicted test data to your training data.\"
\nIn such case you can define a high confidence prediction as one closer enough to one at the final softmax layer. But how to define it in a regression problem like this one?
https://arxiv.org/pdf/2206.14486.pdf
\nNot specific to this dataset/problem, but good paper to think about the data.
", "votes": null, "replies": [ { "id": 2541219, "author_name": "sacuscreed", "author_url": "", "post_date": "11/28/2023 09:55:13", "content": "Thanks, I'll check it.
", "votes": null, "replies": [] } ] } ] }, { "id": 2541160, "author_name": "enriquezaf", "author_url": "", "post_date": "11/28/2023 09:22:42", "content": "It depends in how good are you models at predicting, adding pseudo-labeling got me from 0.15092 to 0.14791 in single model, same architecture/hypers, took 6 times longer (2d 11h 57m 32s) to train though, that was using the pseudo-label + base data, need to finetune with the base data another one trained only with pseudo-labels to see if it makes any difference.
", "votes": null, "replies": [] }, { "id": 2541286, "author_name": "dhruvdhilla", "author_url": "", "post_date": "11/28/2023 11:08:06", "content": "Pseudo lables will only help once you generalize \"good enough\" for longer sequences. If you have bad generalization they will only add noise, which is why i don't think its been talked about.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2540860": "Based on the [Stanford OpenVaccine competition](https://www.kaggle.com/competitions/stanford-covid-vaccine/overview), it seems pseudo-labeling played a big part in many of the top solutions. However, since not many people are talking about it here, is it not likely to provide a big boost in this competition? Has anyone tried it?", "2541125": "From https://www.kaggle.com/code/cdeotte/pseudo-labeling-qda-0-969 \"Pseudo labeling is the process of adding confident predicted test data to your training data.\"\nIn such case you can define a high confidence prediction as one closer enough to one at the final softmax layer. But how to define it in a regression problem like this one?", "2541160": "It depends in how good are you models at predicting, adding pseudo-labeling got me from 0.15092 to 0.14791 in single model, same architecture/hypers, took 6 times longer (2d 11h 57m 32s) to train though, that was using the pseudo-label + base data, need to finetune with the base data another one trained only with pseudo-labels to see if it makes any difference.", "2541190": "https://arxiv.org/pdf/2206.14486.pdf\n\nNot specific to this dataset/problem, but good paper to think about the data.", "2541219": "Thanks, I'll check it.", "2541286": "Pseudo lables will only help once you generalize \"good enough\" for longer sequences. If you have bad generalization they will only add noise, which is why i don't think its been talked about." }, "source": "meta" }