I tried pseudo-labeling with a single se-resnext50 model. My CV score was 0.8745 and LB 0.885. Pseudo-labeling pushed my LB to 0.888. Although it's not that significant(probably because I trained for only a couple of epochs) but I think, with proper tuning, I can push it more.
I tried pseudo-labeling with a single se-resnext50 model. My CV score was 0.8745 and LB 0.885. Pseudo-labeling pushed my LB to 0.888. Although it's not that significant(probably because I trained for only a couple of epochs) but I think, with proper tuning, I can push it more.
Do you have any good resources to understand pseudo-labeling? I have heard about it in many places
", "rawMarkdown": "Do you have any good resources to understand pseudo-labeling? I have heard about it in many places", "votes": null }, { "id": "1093556", "postDate": "11/27/2020 20:34:04", "content": "Pseudo-labeling what? All the data is labeled, right? Are you relabeling data?
", "rawMarkdown": "Pseudo-labeling what? All the data is labeled, right? Are you relabeling data?", "votes": null }, { "id": "1093571", "postDate": "11/27/2020 20:54:00", "content": "@tanlikesmath I'm labeling the test data and then training over the train + newly labeled test data for a couple of epochs during submission.
", "rawMarkdown": "tanlikesmath I'm labeling the test data and then training over the train + newly labeled test data for a couple of epochs during submission.", "votes": null }, { "id": "1093575", "postDate": "11/27/2020 20:55:58", "content": "For example, You can see this notebook from Global Wheat Detection challenge.
", "rawMarkdown": "For example, You can see [this](https://www.kaggle.com/nvnnghia/yolov5-pseudo-labeling ) notebook from **Global Wheat Detection** challenge.", "votes": null }, { "id": "1093580", "postDate": "11/27/2020 21:01:53", "content": "In layman's terms, pseudo-labeling means labeling unseen/test data and then training over them again. Let's say, you have 500 train images, and 100 test images. First, train a model over the train images(labels are known). Then use this model to generate prediction over the 100 test images(the actual label is unknown). Let's say, the confidence score for 70 images are very high. So, include these 70 images into your train data. Consider model predictions for these images as your label. These are called pseudo-labeled data. Then train your model over 500(train)+70(pseudo labeled) data again.
Thanks for the amazing explanation. This sounds cool
", "rawMarkdown": "Thanks for the amazing explanation. This sounds cool", "votes": null }, { "id": "1093651", "postDate": "11/27/2020 22:10:43", "content": "Thank you for the clarification. I was thinking about doing something like this, so I am glad to hear it improved your results and I will be sure to give it a try!
", "rawMarkdown": "Thank you for the clarification. I was thinking about doing something like this, so I am glad to hear it improved your results and I will be sure to give it a try!", "votes": null }, { "id": "1093703", "postDate": "11/28/2020 00:15:25", "content": "I would check out Chris Deotte's notebook on pseudo labeling here - it is an excellent resource.
", "rawMarkdown": "I would check out [Chris Deotte'](https://www.kaggle.com/cdeotte)s notebook on pseudo labeling [here](https://www.kaggle.com/cdeotte/pseudo-labeling-qda-0-969) - it is an excellent resource.", "votes": null }, { "id": "1093900", "postDate": "11/28/2020 05:33:01", "content": "Please go ahead with your experiments and share the results.
", "rawMarkdown": "Please go ahead with your experiments and share the results.", "votes": null }, { "id": "1093929", "postDate": "11/28/2020 06:09:49", "content": "A single mode inference takes 4-5 mins to complete during submission for me. So you can do a lot of things within the 9 hours submission time limit. You can use TTA, pseudo-labelling for K-fold models and ensemble.
", "rawMarkdown": "A single mode inference takes 4-5 mins to complete during submission for me. So you can do a lot of things within the 9 hours submission time limit. You can use TTA, pseudo-labelling for K-fold models and ensemble.", "votes": null }, { "id": "1094052", "postDate": "11/28/2020 09:18:13", "content": "I have higher CV but lower LB with plabelš§
", "rawMarkdown": "I have higher CV but lower LB with plabelš§", "votes": null }, { "id": "1094206", "postDate": "11/28/2020 12:09:48", "content": "Maybe your model is generating too much label noise. I am also quite not sure how my plabel scheme is performing on the private test set.
", "rawMarkdown": "Maybe your model is generating too much label noise. I am also quite not sure how my plabel scheme is performing on the private test set.", "votes": null }, { "id": "1094286", "postDate": "11/28/2020 13:12:50", "content": "Update:
\nTried Pseudo Label + TTA with Effnet B4
\nCV: 0.878
\nLB: 0.882(before) and 0.890(after plabel+TTA)
I think pseudo labelling need some sort of Semi-supervised/Active learning and careful handelling of noisy labels in 2nd stage training to work well in this dataset.
\nI read the notebook by Chris. I don't think his \"passive\" pseudo labelling will work well here.
", "rawMarkdown": "I think pseudo labelling need some sort of Semi-supervised/Active learning and careful handelling of noisy labels in 2nd stage training to work well in this dataset.\n\nI read the notebook by Chris. I don't think his \"passive\" pseudo labelling will work well here.", "votes": null }, { "id": "1094542", "postDate": "11/28/2020 17:52:18", "content": "What do you mean by \"passive\" pseudo labeling?
", "rawMarkdown": "What do you mean by \"passive\" pseudo labeling?", "votes": null }, { "id": "1094549", "postDate": "11/28/2020 18:01:45", "content": "Just inference on unlabelled data and feed again the network with newly labelled dataset.
\nYou may have many noisy labels.
", "rawMarkdown": "Just inference on unlabelled data and feed again the network with newly labelled dataset. \n\nYou may have many noisy labels.", "votes": null }, { "id": "1094566", "postDate": "11/28/2020 18:19:50", "content": "I agree. You have to be careful with handling the new labels. But if your model is good enough, then you can expect that False Negative/False positive cases will be very low in your dataset. Sometimes a little label noise may even help reduce overfitting.
", "rawMarkdown": "I agree. You have to be careful with handling the new labels. But if your model is good enough, then you can expect that False Negative/False positive cases will be very low in your dataset. Sometimes a little label noise may even help reduce overfitting.", "votes": null }, { "id": "1094680", "postDate": "11/28/2020 20:08:01", "content": "Thank you for a nice discussion!
\nDoes (before) mean with TTA or without TTA?
\nDid you try just with pseudo label?
before means no plabel and no TTA.
", "rawMarkdown": "before means no plabel and no TTA.", "votes": null }, { "id": "1095148", "postDate": "11/29/2020 09:48:07", "content": "@Serigne The problem can be pretty much solved if you only take the samples where model confidence is above a certain threshold say 0.95 is a common one.
", "rawMarkdown": "Serigne The problem can be pretty much solved if you only take the samples where model confidence is above a certain threshold say 0.95 is a common one.", "votes": null }, { "id": "1096163", "postDate": "11/30/2020 09:14:37", "content": "Where did you get the test data? I saw in the data download page and the /kaggle/input/cassava-leaf-disease-classification/test_images , both of them have only one img.
", "rawMarkdown": "Where did you get the test data? I saw in the data download page and the /kaggle/input/cassava-leaf-disease-classification/test_images , both of them have only one img.", "votes": null }, { "id": "1096694", "postDate": "11/30/2020 17:43:36", "content": "For this competition, test data is 'unseen' which means you can not see them and can only access them while making a submission through a notebook. The unseen test data contains around 15k images. For more details, you can see this.
", "rawMarkdown": "For this competition, test data is 'unseen' which means you can not see them and can only access them while making a submission through a notebook. The unseen test data contains around 15k images. For more details, you can see [this](https://www.kaggle.com/docs/competitions).", "votes": null }, { "id": "1116068", "postDate": "12/16/2020 19:52:59", "content": "@tahsin how we can calculate the confidence score for pseudo labeled images ?
", "rawMarkdown": "tahsin how we can calculate the confidence score for pseudo labeled images ?", "votes": null }, { "id": "1116114", "postDate": "12/16/2020 21:20:28", "content": "Confidence score means how close the predictions are to \"1\". Let's say your model's prediction over an image is [0.7, 0.1, 0.05, 0.02, 0.13] which means it is 70% confident that the class would be 0. You can calculate these scores for pseudo-labeled images but you won't be able to see them in this competition.
@tahsin I am still confused about how we will decide the threshold for confidence score. For example, lets say my model predictions for an image1 is [0.7, 0.1, 0.05, 0.02, 0.13] -> 70% confident for class 0 and for image2 is [0.2, 0.6, 0.05, 0.02, 0.13] -> 60% confident for class 1 and for image3 is [0.2, 0.2, 0.4, 0.07, 0.13] -> 40% confident for class 2. Now should I consider all 3 images for pseudo labelling or should I select some threshold for confidence score as we can see that in the third example, the confidence score is low(40% only).
Ofc, filter out the images with confidence, a good threshold is where model is 95% confident on one class.
", "rawMarkdown": "Ofc, filter out the images with confidence, a good threshold is where model is 95% confident on one class.", "votes": null }, { "id": "1117139", "postDate": "12/17/2020 18:47:50", "content": "Don't you think with this much high confidence, we will end up with very few images which will not be much useful to increase the model performance.
", "rawMarkdown": "Don't you think with this much high confidence, we will end up with very few images which will not be much useful to increase the model performance.", "votes": null }, { "id": "1117151", "postDate": "12/17/2020 19:04:06", "content": "Ofcourse it will depend how many images you want, but small number of images will help model and more images but some with wrong labels, will only help you make your whole pipeline ready for noise. For a normal model, 95% is the confidence to go, but if you can handle noise ofc you can go for something which chooses more images.
", "rawMarkdown": "Ofcourse it will depend how many images you want, but small number of images will help model and more images but some with wrong labels, will only help you make your whole pipeline ready for noise. For a normal model, 95% is the confidence to go, but if you can handle noise ofc you can go for something which chooses more images.", "votes": null }, { "id": "1120621", "postDate": "12/21/2020 01:39:54", "content": "Did your Pseudo-labeling train all the data, then access the test image and make predictions and add it to the training data?
", "rawMarkdown": "Did your Pseudo-labeling train all the data, then access the test image and make predictions and add it to the training data?", "votes": null }, { "id": "1187784", "postDate": "02/05/2021 17:54:20", "content": "@tahsin can able to understand the pseudo Labelling part clearly but with your explanation,above we have only one given sample of test data then what is the use of using pseudo labelling or how to do it ?
", "rawMarkdown": "tahsin can able to understand the pseudo Labelling part clearly but with your explanation,above we have only one given sample of test data then what is the use of using pseudo labelling or how to do it ?", "votes": null } ], "comments": [ { "id": 1093308, "author_name": "debarshichanda", "author_url": "", "post_date": "11/27/2020 15:52:16", "content": "Do you have any good resources to understand pseudo-labeling? I have heard about it in many places
", "votes": null, "replies": [ { "id": 1093580, "author_name": "tahsin", "author_url": "", "post_date": "11/27/2020 21:01:53", "content": "In layman's terms, pseudo-labeling means labeling unseen/test data and then training over them again. Let's say, you have 500 train images, and 100 test images. First, train a model over the train images(labels are known). Then use this model to generate prediction over the 100 test images(the actual label is unknown). Let's say, the confidence score for 70 images are very high. So, include these 70 images into your train data. Consider model predictions for these images as your label. These are called pseudo-labeled data. Then train your model over 500(train)+70(pseudo labeled) data again.
Thanks for the amazing explanation. This sounds cool
", "votes": null, "replies": [] }, { "id": 1093703, "author_name": "tuckerarrants", "author_url": "", "post_date": "11/28/2020 00:15:25", "content": "I would check out Chris Deotte's notebook on pseudo labeling here - it is an excellent resource.
", "votes": null, "replies": [] }, { "id": 1116068, "author_name": "abdurrehman245", "author_url": "", "post_date": "12/16/2020 19:52:59", "content": "@tahsin how we can calculate the confidence score for pseudo labeled images ?
", "votes": null, "replies": [] }, { "id": 1116114, "author_name": "tahsin", "author_url": "", "post_date": "12/16/2020 21:20:28", "content": "Confidence score means how close the predictions are to \"1\". Let's say your model's prediction over an image is [0.7, 0.1, 0.05, 0.02, 0.13] which means it is 70% confident that the class would be 0. You can calculate these scores for pseudo-labeled images but you won't be able to see them in this competition.
@tahsin I am still confused about how we will decide the threshold for confidence score. For example, lets say my model predictions for an image1 is [0.7, 0.1, 0.05, 0.02, 0.13] -> 70% confident for class 0 and for image2 is [0.2, 0.6, 0.05, 0.02, 0.13] -> 60% confident for class 1 and for image3 is [0.2, 0.2, 0.4, 0.07, 0.13] -> 40% confident for class 2. Now should I consider all 3 images for pseudo labelling or should I select some threshold for confidence score as we can see that in the third example, the confidence score is low(40% only).
Ofc, filter out the images with confidence, a good threshold is where model is 95% confident on one class.
", "votes": null, "replies": [] }, { "id": 1117139, "author_name": "abdurrehman245", "author_url": "", "post_date": "12/17/2020 18:47:50", "content": "Don't you think with this much high confidence, we will end up with very few images which will not be much useful to increase the model performance.
", "votes": null, "replies": [] }, { "id": 1117151, "author_name": "harshitsheoran", "author_url": "", "post_date": "12/17/2020 19:04:06", "content": "Ofcourse it will depend how many images you want, but small number of images will help model and more images but some with wrong labels, will only help you make your whole pipeline ready for noise. For a normal model, 95% is the confidence to go, but if you can handle noise ofc you can go for something which chooses more images.
", "votes": null, "replies": [] }, { "id": 1187784, "author_name": "vpkprasanna", "author_url": "", "post_date": "02/05/2021 17:54:20", "content": "@tahsin can able to understand the pseudo Labelling part clearly but with your explanation,above we have only one given sample of test data then what is the use of using pseudo labelling or how to do it ?
", "votes": null, "replies": [] } ] }, { "id": 1093556, "author_name": "tanlikesmath", "author_url": "", "post_date": "11/27/2020 20:34:04", "content": "Pseudo-labeling what? All the data is labeled, right? Are you relabeling data?
", "votes": null, "replies": [ { "id": 1093571, "author_name": "tahsin", "author_url": "", "post_date": "11/27/2020 20:54:00", "content": "@tanlikesmath I'm labeling the test data and then training over the train + newly labeled test data for a couple of epochs during submission.
", "votes": null, "replies": [] }, { "id": 1093575, "author_name": "tahsin", "author_url": "", "post_date": "11/27/2020 20:55:58", "content": "For example, You can see this notebook from Global Wheat Detection challenge.
", "votes": null, "replies": [] }, { "id": 1093651, "author_name": "tanlikesmath", "author_url": "", "post_date": "11/27/2020 22:10:43", "content": "Thank you for the clarification. I was thinking about doing something like this, so I am glad to hear it improved your results and I will be sure to give it a try!
", "votes": null, "replies": [] }, { "id": 1093900, "author_name": "tahsin", "author_url": "", "post_date": "11/28/2020 05:33:01", "content": "Please go ahead with your experiments and share the results.
", "votes": null, "replies": [] }, { "id": 1096163, "author_name": "fire15", "author_url": "", "post_date": "11/30/2020 09:14:37", "content": "Where did you get the test data? I saw in the data download page and the /kaggle/input/cassava-leaf-disease-classification/test_images , both of them have only one img.
", "votes": null, "replies": [] }, { "id": 1096694, "author_name": "tahsin", "author_url": "", "post_date": "11/30/2020 17:43:36", "content": "For this competition, test data is 'unseen' which means you can not see them and can only access them while making a submission through a notebook. The unseen test data contains around 15k images. For more details, you can see this.
", "votes": null, "replies": [] } ] }, { "id": 1093929, "author_name": "tahsin", "author_url": "", "post_date": "11/28/2020 06:09:49", "content": "A single mode inference takes 4-5 mins to complete during submission for me. So you can do a lot of things within the 9 hours submission time limit. You can use TTA, pseudo-labelling for K-fold models and ensemble.
", "votes": null, "replies": [] }, { "id": 1094052, "author_name": "dldmw579", "author_url": "", "post_date": "11/28/2020 09:18:13", "content": "I have higher CV but lower LB with plabelš§
", "votes": null, "replies": [ { "id": 1094206, "author_name": "tahsin", "author_url": "", "post_date": "11/28/2020 12:09:48", "content": "Maybe your model is generating too much label noise. I am also quite not sure how my plabel scheme is performing on the private test set.
", "votes": null, "replies": [] } ] }, { "id": 1094286, "author_name": "tahsin", "author_url": "", "post_date": "11/28/2020 13:12:50", "content": "Update:
\nTried Pseudo Label + TTA with Effnet B4
\nCV: 0.878
\nLB: 0.882(before) and 0.890(after plabel+TTA)
Thank you for a nice discussion!
\nDoes (before) mean with TTA or without TTA?
\nDid you try just with pseudo label?
before means no plabel and no TTA.
", "votes": null, "replies": [] } ] }, { "id": 1094416, "author_name": "serigne", "author_url": "", "post_date": "11/28/2020 15:48:20", "content": "I think pseudo labelling need some sort of Semi-supervised/Active learning and careful handelling of noisy labels in 2nd stage training to work well in this dataset.
\nI read the notebook by Chris. I don't think his \"passive\" pseudo labelling will work well here.
", "votes": null, "replies": [ { "id": 1094542, "author_name": "tahsin", "author_url": "", "post_date": "11/28/2020 17:52:18", "content": "What do you mean by \"passive\" pseudo labeling?
", "votes": null, "replies": [] }, { "id": 1094549, "author_name": "serigne", "author_url": "", "post_date": "11/28/2020 18:01:45", "content": "Just inference on unlabelled data and feed again the network with newly labelled dataset.
\nYou may have many noisy labels.
", "votes": null, "replies": [] }, { "id": 1094566, "author_name": "tahsin", "author_url": "", "post_date": "11/28/2020 18:19:50", "content": "I agree. You have to be careful with handling the new labels. But if your model is good enough, then you can expect that False Negative/False positive cases will be very low in your dataset. Sometimes a little label noise may even help reduce overfitting.
", "votes": null, "replies": [] }, { "id": 1095148, "author_name": "harshitsheoran", "author_url": "", "post_date": "11/29/2020 09:48:07", "content": "@Serigne The problem can be pretty much solved if you only take the samples where model confidence is above a certain threshold say 0.95 is a common one.
", "votes": null, "replies": [] } ] }, { "id": 1120621, "author_name": "sjtuyxc", "author_url": "", "post_date": "12/21/2020 01:39:54", "content": "Did your Pseudo-labeling train all the data, then access the test image and make predictions and add it to the training data?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1092933": "I tried pseudo-labeling with a single `se-resnext50` model. My CV score was `0.8745` and LB `0.885`. Pseudo-labeling pushed my LB to `0.888`. Although it's not that significant(probably because I trained for only a couple of epochs) but I think, with proper tuning, I can push it more.", "1093308": "Do you have any good resources to understand pseudo-labeling? I have heard about it in many places", "1093556": "Pseudo-labeling what? All the data is labeled, right? Are you relabeling data?", "1093571": "tanlikesmath I'm labeling the test data and then training over the train + newly labeled test data for a couple of epochs during submission.", "1093575": "For example, You can see [this](https://www.kaggle.com/nvnnghia/yolov5-pseudo-labeling ) notebook from **Global Wheat Detection** challenge.", "1093580": "In layman's terms, pseudo-labeling means labeling unseen/test data and then training over them again. Let's say, you have 500 train images, and 100 test images. First, train a model over the train images(labels are known). Then use this model to generate prediction over the 100 test images(the actual label is unknown). Let's say, the confidence score for 70 images are very high. So, include these 70 images into your train data. Consider model predictions for these images as your label. These are called `pseudo-labeled` data. Then train your model over 500(train)+70(pseudo labeled) data again.", "1093594": "Thanks for the amazing explanation. This sounds cool", "1093651": "Thank you for the clarification. I was thinking about doing something like this, so I am glad to hear it improved your results and I will be sure to give it a try!", "1093703": "I would check out [Chris Deotte'](https://www.kaggle.com/cdeotte)s notebook on pseudo labeling [here](https://www.kaggle.com/cdeotte/pseudo-labeling-qda-0-969) - it is an excellent resource.", "1093900": "Please go ahead with your experiments and share the results.", "1093929": "A single mode inference takes 4-5 mins to complete during submission for me. So you can do a lot of things within the 9 hours submission time limit. You can use TTA, pseudo-labelling for K-fold models and ensemble.", "1094052": "I have higher CV but lower LB with plabelš§", "1094206": "Maybe your model is generating too much label noise. I am also quite not sure how my plabel scheme is performing on the private test set.", "1094286": "Update:\nTried Pseudo Label + TTA with Effnet B4\nCV: 0.878\nLB: 0.882(before) and 0.890(after plabel+TTA)", "1094416": "I think pseudo labelling need some sort of Semi-supervised/Active learning and careful handelling of noisy labels in 2nd stage training to work well in this dataset.\n\nI read the notebook by Chris. I don't think his \"passive\" pseudo labelling will work well here.", "1094542": "What do you mean by \"passive\" pseudo labeling?", "1094549": "Just inference on unlabelled data and feed again the network with newly labelled dataset. \n\nYou may have many noisy labels.", "1094566": "I agree. You have to be careful with handling the new labels. But if your model is good enough, then you can expect that False Negative/False positive cases will be very low in your dataset. Sometimes a little label noise may even help reduce overfitting.", "1094680": "Thank you for a nice discussion!\nDoes (before) mean with TTA or without TTA?\nDid you try just with pseudo label?", "1095002": "before means no plabel and no TTA.", "1095148": "Serigne The problem can be pretty much solved if you only take the samples where model confidence is above a certain threshold say 0.95 is a common one.", "1096163": "Where did you get the test data? I saw in the data download page and the /kaggle/input/cassava-leaf-disease-classification/test_images , both of them have only one img.", "1096694": "For this competition, test data is 'unseen' which means you can not see them and can only access them while making a submission through a notebook. The unseen test data contains around 15k images. For more details, you can see [this](https://www.kaggle.com/docs/competitions).", "1116068": "tahsin how we can calculate the confidence score for pseudo labeled images ?", "1116114": "Confidence score means how close the predictions are to \"1\". Let's say your model's prediction over an image is `[0.7, 0.1, 0.05, 0.02, 0.13]` which means it is 70% confident that the class would be 0. You can calculate these scores for pseudo-labeled images but you won't be able to see them in this competition.", "1117097": "tahsin I am still confused about how we will decide the threshold for confidence score. For example, lets say my model predictions for an image1 is `[0.7, 0.1, 0.05, 0.02, 0.13] -> 70% confident for class 0` and for image2 is `[0.2, 0.6, 0.05, 0.02, 0.13] -> 60% confident for class 1` and for image3 is `[0.2, 0.2, 0.4, 0.07, 0.13] -> 40% confident for class 2`. Now should I consider all 3 images for pseudo labelling or should I select some threshold for confidence score as we can see that in the third example, the confidence score is low(40% only).", "1117111": "Ofc, filter out the images with confidence, a good threshold is where model is 95% confident on one class.", "1117139": "Don't you think with this much high confidence, we will end up with very few images which will not be much useful to increase the model performance.", "1117151": "Ofcourse it will depend how many images you want, but small number of images will help model and more images but some with wrong labels, will only help you make your whole pipeline ready for noise. For a normal model, 95% is the confidence to go, but if you can handle noise ofc you can go for something which chooses more images.", "1120621": "Did your Pseudo-labeling train all the data, then access the test image and make predictions and add it to the training data?", "1187784": "tahsin can able to understand the pseudo Labelling part clearly but with your explanation,above we have only one given sample of test data then what is the use of using pseudo labelling or how to do it ?" }, "source": "meta" }