I found this package while searching ways to deal with noisy data.
\ngithub: https://github.com/cgnorthcutt/cleanlab
\ndetail: https://l7.curtisnorthcutt.com/confident-learning
\npaper: https://arxiv.org/abs/1911.00068
cleanlab is a machine learning python package for learning with noisy labels and finding label errors in datasets. cleanlab CLEANs LABels. It is powered by the theory of confident learning, published in this paper | blog.
I have made a dataset and demo notebook.
\ndataset: https://www.kaggle.com/tmhrkt/cleanlab
\nnotebook: https://www.kaggle.com/tmhrkt/cassava-cleanlab-with-efficientnet-b0
\nI hope it helps you :)
I found this package while searching ways to deal with noisy data.
\ngithub: https://github.com/cgnorthcutt/cleanlab
\ndetail: https://l7.curtisnorthcutt.com/confident-learning
\npaper: https://arxiv.org/abs/1911.00068
cleanlab is a machine learning python package for learning with noisy labels and finding label errors in datasets. cleanlab CLEANs LABels. It is powered by the theory of confident learning, published in this paper | blog.
I have made a dataset and demo notebook.
\ndataset: https://www.kaggle.com/tmhrkt/cleanlab
\nnotebook: https://www.kaggle.com/tmhrkt/cassava-cleanlab-with-efficientnet-b0
\nI hope it helps you :)
The simplest way to clean up the false labels is to predict the training set and verification set again. If the confidence level is lower than a certain level, it will be judged as false labels. But this confidence is not easy to grasp, too much cleaning will lose important features, too little cleaning, the final effect of training image is very little. I remember that I have done similar work for this project before, at least 300-400 images have been cleaned up, and at most about 1000 images have been cleaned up, but all of them failed to achieve good results. Since the image noise of training set, verification set and test set is noisy, their distribution is similar. What should we do if the private test set also has error labels similar to the training set? So cleaning up the label may reduce the lb. of course, if the private test set is clean, the noise is small, and there are few wrong labels, then cleaning up the training set label is useful.
", "rawMarkdown": "The simplest way to clean up the false labels is to predict the training set and verification set again. If the confidence level is lower than a certain level, it will be judged as false labels. But this confidence is not easy to grasp, too much cleaning will lose important features, too little cleaning, the final effect of training image is very little. I remember that I have done similar work for this project before, at least 300-400 images have been cleaned up, and at most about 1000 images have been cleaned up, but all of them failed to achieve good results. Since the image noise of training set, verification set and test set is noisy, their distribution is similar. What should we do if the private test set also has error labels similar to the training set? So cleaning up the label may reduce the lb. of course, if the private test set is clean, the noise is small, and there are few wrong labels, then cleaning up the training set label is useful.", "votes": 5 }, { "id": 1169294, "postDate": "2021-01-25T12:21:07.143Z", "content": "@zhangeng i agree with your thoughts, i think we should let model fit the noisy label insteading of cleaning up the noisy label
", "rawMarkdown": "@zhangeng i agree with your thoughts, i think we should let model fit the noisy label insteading of cleaning up the noisy label" }, { "id": 1154427, "postDate": "2021-01-15T16:20:46.050Z", "content": "Hey! I'm trying to implement it, but if my understanding is correct i'm training a classifier on top a classifier right? How would i go about saving this cleanlab model and then loading it for inference in the test set?
", "rawMarkdown": "Hey! I'm trying to implement it, but if my understanding is correct i'm training a classifier on top a classifier right? How would i go about saving this cleanlab model and then loading it for inference in the test set?", "replies": [ { "id": 1155111, "postDate": "2021-01-16T08:04:02.423Z", "content": "You can load and predict from pretrained weights by doing this
\ncassava_model = CassavaImgClassifier(params[\"model_name\"], pretrained=False)\ncassava_model.load_state_dict(torch.load(\"path/to/pretrained/weights.pth\"))\nclf = Classifier(cassava_model, params)\npredictions = clf.predict_proba(sample_submission.loc[:, \"image_id\"].values, phase=\"test\")\nThanks for your reply, but i'm still wondering about how you get the weights from the cleanlab classifier to a file so that i can then load it into another notebook using your code.
\nThanks in advance!
", "rawMarkdown": "Thanks for your reply, but i'm still wondering about how you get the weights from the cleanlab classifier to a file so that i can then load it into another notebook using your code.\n\nThanks in advance!" }, { "id": 1156270, "postDate": "2021-01-17T03:59:36.587Z", "content": "LearningWithNoisyLabels returns LearningWithNoisyLabels.clf, which is Classifier class with trained model in my notebook. Classifier class has model, which is pytorch model. So you can get the trained weights from Classifier.model.state_dict() or LearningWithNoisyLabels.clf.model.state_dict() and save them. Then, you can upload the trained weights to your own dataset and load them in your inference notebook.
Hey thanks for the good work.
\nDoes the dataset contain a csv file with cleaned labels?
No. The dataset is just a copy of github repository.
", "rawMarkdown": "No. The dataset is just a copy of [github](https://github.com/cgnorthcutt/cleanlab) repository. " } ] }, { "id": 1151437, "postDate": "2021-01-13T10:23:22.237Z", "content": "Really interesting. Thanks for sharing. What about performance and accuracy?
", "rawMarkdown": "Really interesting. Thanks for sharing. What about performance and accuracy? ", "replies": [ { "id": 1151461, "postDate": "2021-01-13T10:44:11.313Z", "content": "The performance is better than the model without cleanlab. See my comment in this discussion :)
Thanks for sharing! This seems very interesting. Have you tested this on your submissions yet? How were the results?
", "rawMarkdown": "Thanks for sharing! This seems very interesting. Have you tested this on your submissions yet? How were the results?", "replies": [ { "id": 1148945, "postDate": "2021-01-11T13:45:36.743Z", "content": "I'm now training. I'll update soon.
", "rawMarkdown": "I'm now training. I'll update soon.", "votes": 1 }, { "id": 1150085, "postDate": "2021-01-12T11:04:10.670Z", "content": "Training with cleanlab,
\nCV: 0.8808
\nLB: 0.885
Training without cleanlab,
\nCV: 0.8786
\nLB: 0.882
They are the same condition except using cleanlab.
", "rawMarkdown": "Training with cleanlab,\nCV: 0.8808\nLB: 0.885\n\nTraining without cleanlab,\nCV: 0.8786\nLB: 0.882 \n\nThey are the same condition except using cleanlab. ", "votes": 4 }, { "id": 1150355, "postDate": "2021-01-12T14:28:46.737Z", "content": "Is that single model single fold or an ensemble? Thanks for sharing!
", "rawMarkdown": "Is that single model single fold or an ensemble? Thanks for sharing!" }, { "id": 1150942, "postDate": "2021-01-13T01:37:42.487Z", "content": "It is 5 fold averaging with one seed. Parameters are the same as the sharing notebook.
", "rawMarkdown": "It is 5 fold averaging with one seed. Parameters are the same as the sharing notebook.", "votes": 1 }, { "id": 1155144, "postDate": "2021-01-16T08:28:07.440Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 1152447, "postDate": "2021-01-14T06:53:46.857Z", "content": "Thanks for sharing !!! @tmhrkt
", "rawMarkdown": "Thanks for sharing !!! @tmhrkt ", "votes": 1 } ], "comments": [ { "id": 1151094, "author_name": "zhangeng", "author_url": "", "post_date": "2021-01-13T06:17:47.927000", "content": "The simplest way to clean up the false labels is to predict the training set and verification set again. If the confidence level is lower than a certain level, it will be judged as false labels. But this confidence is not easy to grasp, too much cleaning will lose important features, too little cleaning, the final effect of training image is very little. I remember that I have done similar work for this project before, at least 300-400 images have been cleaned up, and at most about 1000 images have been cleaned up, but all of them failed to achieve good results. Since the image noise of training set, verification set and test set is noisy, their distribution is similar. What should we do if the private test set also has error labels similar to the training set? So cleaning up the label may reduce the lb. of course, if the private test set is clean, the noise is small, and there are few wrong labels, then cleaning up the training set label is useful.
", "votes": 5, "replies": [] }, { "id": 1169294, "author_name": "Jackie Mai", "author_url": "", "post_date": "2021-01-25T12:21:07.143000", "content": "@zhangeng i agree with your thoughts, i think we should let model fit the noisy label insteading of cleaning up the noisy label
", "votes": 0, "replies": [] }, { "id": 1154427, "author_name": "Gabriel Prado", "author_url": "", "post_date": "2021-01-15T16:20:46.050000", "content": "Hey! I'm trying to implement it, but if my understanding is correct i'm training a classifier on top a classifier right? How would i go about saving this cleanlab model and then loading it for inference in the test set?
", "votes": 0, "replies": [ { "id": 1155111, "author_name": "ktm", "author_url": "", "post_date": "2021-01-16T08:04:02.423000", "content": "You can load and predict from pretrained weights by doing this
\ncassava_model = CassavaImgClassifier(params[\"model_name\"], pretrained=False)\ncassava_model.load_state_dict(torch.load(\"path/to/pretrained/weights.pth\"))\nclf = Classifier(cassava_model, params)\npredictions = clf.predict_proba(sample_submission.loc[:, \"image_id\"].values, phase=\"test\")\nThanks for your reply, but i'm still wondering about how you get the weights from the cleanlab classifier to a file so that i can then load it into another notebook using your code.
\nThanks in advance!
", "votes": 0, "replies": [] }, { "id": 1156270, "author_name": "ktm", "author_url": "", "post_date": "2021-01-17T03:59:36.587000", "content": "LearningWithNoisyLabels returns LearningWithNoisyLabels.clf, which is Classifier class with trained model in my notebook. Classifier class has model, which is pytorch model. So you can get the trained weights from Classifier.model.state_dict() or LearningWithNoisyLabels.clf.model.state_dict() and save them. Then, you can upload the trained weights to your own dataset and load them in your inference notebook.
Hey thanks for the good work.
\nDoes the dataset contain a csv file with cleaned labels?
No. The dataset is just a copy of github repository.
", "votes": 0, "replies": [] } ] }, { "id": 1151437, "author_name": "Md. Masud Rana", "author_url": "", "post_date": "2021-01-13T10:23:22.237000", "content": "Really interesting. Thanks for sharing. What about performance and accuracy?
", "votes": 0, "replies": [ { "id": 1151461, "author_name": "ktm", "author_url": "", "post_date": "2021-01-13T10:44:11.313000", "content": "The performance is better than the model without cleanlab. See my comment in this discussion :)
Thanks for sharing! This seems very interesting. Have you tested this on your submissions yet? How were the results?
", "votes": 0, "replies": [ { "id": 1148945, "author_name": "ktm", "author_url": "", "post_date": "2021-01-11T13:45:36.743000", "content": "I'm now training. I'll update soon.
", "votes": 1, "replies": [] }, { "id": 1150085, "author_name": "ktm", "author_url": "", "post_date": "2021-01-12T11:04:10.670000", "content": "Training with cleanlab,
\nCV: 0.8808
\nLB: 0.885
Training without cleanlab,
\nCV: 0.8786
\nLB: 0.882
They are the same condition except using cleanlab.
", "votes": 4, "replies": [] }, { "id": 1150355, "author_name": "Gabriel Prado", "author_url": "", "post_date": "2021-01-12T14:28:46.737000", "content": "Is that single model single fold or an ensemble? Thanks for sharing!
", "votes": 0, "replies": [] }, { "id": 1150942, "author_name": "ktm", "author_url": "", "post_date": "2021-01-13T01:37:42.487000", "content": "It is 5 fold averaging with one seed. Parameters are the same as the sharing notebook.
", "votes": 1, "replies": [] }, { "id": 1155144, "author_name": "", "author_url": "", "post_date": "2021-01-16T08:28:07.440000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1152447, "author_name": "Satyajit Pattnaik", "author_url": "", "post_date": "2021-01-14T06:53:46.857000", "content": "Thanks for sharing !!! @tmhrkt
", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "1148645": "I found this package while searching ways to deal with noisy data. \ngithub: https://github.com/cgnorthcutt/cleanlab \ndetail: https://l7.curtisnorthcutt.com/confident-learning \npaper: https://arxiv.org/abs/1911.00068 \n \n\n``cleanlab`` is a machine learning python package for **learning with noisy labels** and **finding label errors in datasets**. ``cleanlab`` CLEANs LABels. It is powered by the theory of **confident learning**, published in this [paper](https://arxiv.org/abs/1911.00068) | [blog](https://l7.curtisnorthcutt.com/confident-learning). \n\n\nI have made a dataset and demo notebook. \ndataset: https://www.kaggle.com/tmhrkt/cleanlab \nnotebook: https://www.kaggle.com/tmhrkt/cassava-cleanlab-with-efficientnet-b0 \nI hope it helps you :)", "1151094": "The simplest way to clean up the false labels is to predict the training set and verification set again. If the confidence level is lower than a certain level, it will be judged as false labels. But this confidence is not easy to grasp, too much cleaning will lose important features, too little cleaning, the final effect of training image is very little. I remember that I have done similar work for this project before, at least 300-400 images have been cleaned up, and at most about 1000 images have been cleaned up, but all of them failed to achieve good results. Since the image noise of training set, verification set and test set is noisy, their distribution is similar. What should we do if the private test set also has error labels similar to the training set? So cleaning up the label may reduce the lb. of course, if the private test set is clean, the noise is small, and there are few wrong labels, then cleaning up the training set label is useful.", "1169294": "@zhangeng i agree with your thoughts, i think we should let model fit the noisy label insteading of cleaning up the noisy label", "1154427": "Hey! I'm trying to implement it, but if my understanding is correct i'm training a classifier on top a classifier right? How would i go about saving this cleanlab model and then loading it for inference in the test set?", "1152138": "Hey thanks for the good work. \nDoes the dataset contain a csv file with cleaned labels?", "1151437": "Really interesting. Thanks for sharing. What about performance and accuracy? ", "1148743": "Thanks for sharing! This seems very interesting. Have you tested this on your submissions yet? How were the results?", "1152447": "Thanks for sharing !!! @tmhrkt " } }