![\"\"](\"https://raw.githubusercontent.com/dimitreOliveira/MachineLearning/master/Kaggle/Cassava%20Leaf%20Disease%20Classification/sscl_vs_scl.png\")
Since I saw the concept of contrastive learning I became very interested in the subject, and a few weeks ago I saw an implementation on the Keras tutorials page, so I decided to try it at this competition, I have spent most of my TPU quota from the last two weeks experimenting with it, and I think it has some real potential, so I want to share my finding, maybe some people can share ideas, suggestions and further improve the results.
\nUpdate: I wrote an article about this Supervised Contrastive Learning for Cassava Leaf Disease Classification.
\n
\n\nClusters of points belonging to the same class are pulled together in embedding space, while simultaneously pushing apart clusters of samples from different classes.
\n
The idea is that training models using the supervised contrastive learning can make the model encoder learn better class representation from the samples, this should lead to better generalization and robustness to image and label corruption
\nTraining notebook: https://www.kaggle.com/dimitreoliveira/cassava-leaf-supervised-contrastive-learning
\nInference notebook: https://www.kaggle.com/dimitreoliveira/cassava-supervised-contrastive-learning-inference
What improved performance
\nLarger batch size
\nAverage encoder size
\nData augmentation
\nLager image resolution
\nUsing pre-trained weights, I tried training from scratch but the results were very poor.
\nOversampling the minority classes from the dataset, gave a little help, but I expected more.
What made no difference
\nTemperature parameter, according to the paper, lower temperature can benefit from longer training, since the data here is limited, tweaking this parameter maybe not worth it.
\nDifferent projection heads, I experimented a little with the number of neurons but got no relevant improvements.
\nDifferent classifier heads, I experimented a little with the number of neurons but got no relevant improvements.
Next steps
\nDifferent pooling heads (AVG+MAX, Multi-sample, etc).
\nDifferent optimizers.
\nMore elaborated data augmentation schedules (MixUp, CutMix, etc).
\nUse external data.
\nIt may be interesting to try doing 2nd stage training but fine-tuning the complete network.
An interesting way to evaluate the learned representation is to visualize the output of the trained embedding, at the training notebook I plot the predictions from the model trained using cross-entropy and supervised contrastive learning
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1182060%2F95d2c3c2071b70bf5a78b842181b584d%2FScreenshot%20from%202020-12-24%2014-51-18.png?generation=1608832461429961&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1182060%2F433c6150356e7a709efc5698cc90e0c6%2FScreenshot%20from%202020-12-24%2014-51-31.png?generation=1608832477794000&alt=media\")
You can see that the representation from the model trained with supervised contrastive learning clusters much more than the model trained with cross-entropy.
Since I saw the concept of contrastive learning I became very interested in the subject, and a few weeks ago I saw an implementation on the Keras tutorials page, so I decided to try it at this competition, I have spent most of my TPU quota from the last two weeks experimenting with it, and I think it has some real potential, so I want to share my finding, maybe some people can share ideas, suggestions and further improve the results.
\nUpdate: I wrote an article about this Supervised Contrastive Learning for Cassava Leaf Disease Classification.
\n
\n\nClusters of points belonging to the same class are pulled together in embedding space, while simultaneously pushing apart clusters of samples from different classes.
\n
The idea is that training models using the supervised contrastive learning can make the model encoder learn better class representation from the samples, this should lead to better generalization and robustness to image and label corruption
\nTraining notebook: https://www.kaggle.com/dimitreoliveira/cassava-leaf-supervised-contrastive-learning
\nInference notebook: https://www.kaggle.com/dimitreoliveira/cassava-supervised-contrastive-learning-inference
What improved performance
\nLarger batch size
\nAverage encoder size
\nData augmentation
\nLager image resolution
\nUsing pre-trained weights, I tried training from scratch but the results were very poor.
\nOversampling the minority classes from the dataset, gave a little help, but I expected more.
What made no difference
\nTemperature parameter, according to the paper, lower temperature can benefit from longer training, since the data here is limited, tweaking this parameter maybe not worth it.
\nDifferent projection heads, I experimented a little with the number of neurons but got no relevant improvements.
\nDifferent classifier heads, I experimented a little with the number of neurons but got no relevant improvements.
Next steps
\nDifferent pooling heads (AVG+MAX, Multi-sample, etc).
\nDifferent optimizers.
\nMore elaborated data augmentation schedules (MixUp, CutMix, etc).
\nUse external data.
\nIt may be interesting to try doing 2nd stage training but fine-tuning the complete network.
An interesting way to evaluate the learned representation is to visualize the output of the trained embedding, at the training notebook I plot the predictions from the model trained using cross-entropy and supervised contrastive learning
You can see that the representation from the model trained with supervised contrastive learning clusters much more than the model trained with cross-entropy.
About the model performance, the models trained with supervised contrastive learning consistently performed better (higher LB score) than the models trained with cross-entropy, I ran out of TPU quota, but I am sure I could push it easily to 0.900 tweaking the parameters a little more.
very interesting
", "rawMarkdown": "very interesting", "votes": null }, { "id": "1126216", "postDate": "12/25/2020 12:43:01", "content": "The approach is definetly interesting but with one caveat. Our data has mislabelled examples (noisy labels) and I am afraid that Supervised Contrastive loss is not robust to noisy labels. It will be interesting to see results with just plain SimCLR(self supervision).
", "rawMarkdown": "The approach is definetly interesting but with one caveat. Our data has mislabelled examples (noisy labels) and I am afraid that Supervised Contrastive loss is not robust to noisy labels. It will be interesting to see results with just plain SimCLR(self supervision).", "votes": null }, { "id": "1126758", "postDate": "12/25/2020 22:39:23", "content": "Hi @atharvap329 , according to the paper, supervised contrastive learning performed better than cross-entropy on a benchmark that evaluate performance of different levels of corruption on imagenet, I am not sure the kind of corruption, but it might be worth it to git it a try.
", "rawMarkdown": "Hi @atharvap329 , according to the paper, supervised contrastive learning performed better than cross-entropy on a benchmark that evaluate performance of different levels of corruption on imagenet, I am not sure the kind of corruption, but it might be worth it to git it a try.", "votes": null }, { "id": "1126818", "postDate": "12/26/2020 01:54:22", "content": "They mention the corruption of images due to image images but its worth a try for sure it may happen that supervised contrastive loss may inherently be actually robust to noise(which will be a new finding :p). Looking forward to your results.
", "rawMarkdown": "They mention the corruption of images due to image images but its worth a try for sure it may happen that supervised contrastive loss may inherently be actually robust to noise(which will be a new finding :p). Looking forward to your results.", "votes": null }, { "id": "1127635", "postDate": "12/26/2020 17:47:49", "content": "Interesting. Need to be looked up :)
", "rawMarkdown": "Interesting. Need to be looked up :)", "votes": null }, { "id": "1128194", "postDate": "12/27/2020 08:46:01", "content": "Intersting. (Y)
", "rawMarkdown": "Intersting. (Y)", "votes": null }, { "id": "1171609", "postDate": "01/27/2021 01:43:43", "content": "For those interested in this method I just wrote an article about it, using this competition's data as a use case
", "rawMarkdown": "For those interested in this method I just wrote [an article about it](https://medium.com/towards-artificial-intelligence/supervised-contrastive-learning-for-cassava-leaf-disease-classification-9dd47779a966), using this competition's data as a use case", "votes": null } ], "comments": [ { "id": 1125445, "author_name": "dimitreoliveira", "author_url": "", "post_date": "12/24/2020 18:00:24", "content": "About the model performance, the models trained with supervised contrastive learning consistently performed better (higher LB score) than the models trained with cross-entropy, I ran out of TPU quota, but I am sure I could push it easily to 0.900 tweaking the parameters a little more.
very interesting
", "votes": null, "replies": [] }, { "id": 1126216, "author_name": "atharvap329", "author_url": "", "post_date": "12/25/2020 12:43:01", "content": "The approach is definetly interesting but with one caveat. Our data has mislabelled examples (noisy labels) and I am afraid that Supervised Contrastive loss is not robust to noisy labels. It will be interesting to see results with just plain SimCLR(self supervision).
", "votes": null, "replies": [ { "id": 1126758, "author_name": "dimitreoliveira", "author_url": "", "post_date": "12/25/2020 22:39:23", "content": "Hi @atharvap329 , according to the paper, supervised contrastive learning performed better than cross-entropy on a benchmark that evaluate performance of different levels of corruption on imagenet, I am not sure the kind of corruption, but it might be worth it to git it a try.
", "votes": null, "replies": [] }, { "id": 1126818, "author_name": "atharvap329", "author_url": "", "post_date": "12/26/2020 01:54:22", "content": "They mention the corruption of images due to image images but its worth a try for sure it may happen that supervised contrastive loss may inherently be actually robust to noise(which will be a new finding :p). Looking forward to your results.
", "votes": null, "replies": [] } ] }, { "id": 1127635, "author_name": "atharvaingle", "author_url": "", "post_date": "12/26/2020 17:47:49", "content": "Interesting. Need to be looked up :)
", "votes": null, "replies": [] }, { "id": 1128194, "author_name": "kiranchaudharyy", "author_url": "", "post_date": "12/27/2020 08:46:01", "content": "Intersting. (Y)
", "votes": null, "replies": [] }, { "id": 1171609, "author_name": "dimitreoliveira", "author_url": "", "post_date": "01/27/2021 01:43:43", "content": "For those interested in this method I just wrote an article about it, using this competition's data as a use case
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1125436": "Since I saw the concept of contrastive learning I became very interested in the subject, and a few weeks ago I saw an implementation on the [Keras tutorials page](https://keras.io/examples/vision/supervised-contrastive-learning/), so I decided to try it at this competition, I have spent most of my TPU quota from the last two weeks experimenting with it, and I think it has some real potential, so I want to share my finding, maybe some people can share ideas, suggestions and further improve the results.\n\nUpdate: I wrote an article about this [Supervised Contrastive Learning for Cassava Leaf Disease Classification](https://medium.com/towards-artificial-intelligence/supervised-contrastive-learning-for-cassava-leaf-disease-classification-9dd47779a966).\n\n## Supervised Contrastive Learning\n\n\n\n#### In short, this is how it works\n> Clusters of points belonging to the same class are pulled together in embedding space, while simultaneously pushing apart clusters of samples from different classes.\n\nThe idea is that training models using the supervised contrastive learning can make the model encoder learn better class representation from the samples, this should lead to better generalization and robustness to image and label corruption\n\n\n**Training notebook:** https://www.kaggle.com/dimitreoliveira/cassava-leaf-supervised-contrastive-learning\n**Inference notebook:** https://www.kaggle.com/dimitreoliveira/cassava-supervised-contrastive-learning-inference\n\n___\n\n### Experiments summary\n\n**What improved performance**\nLarger batch size\nAverage encoder size\nData augmentation\nLager image resolution\nUsing pre-trained weights, I tried training from scratch but the results were very poor.\nOversampling the minority classes from the dataset, gave a little help, but I expected more.\n\n___\n\n**What made no difference**\nTemperature parameter, according to the paper, lower temperature can benefit from longer training, since the data here is limited, tweaking this parameter maybe not worth it.\nDifferent projection heads, I experimented a little with the number of neurons but got no relevant improvements.\nDifferent classifier heads, I experimented a little with the number of neurons but got no relevant improvements.\n\n___\n\n**Next steps**\nDifferent pooling heads (AVG+MAX, Multi-sample, etc).\nDifferent optimizers.\nMore elaborated data augmentation schedules (MixUp, CutMix, etc).\nUse external data.\nIt may be interesting to try doing 2nd stage training but fine-tuning the complete network.\n\n___\n\nAn interesting way to evaluate the learned representation is to visualize the output of the trained embedding, at the [training notebook](https://www.kaggle.com/dimitreoliveira/cassava-leaf-supervised-contrastive-learning#Visualize-embeddings-outputs) I plot the predictions from the model trained using `cross-entropy` and `supervised contrastive learning`\n\n#### Embedding trained with cross-entropy\n\n\n\n#### Embedding trained with supervised contrastive learning\n\n\n\nYou can see that the representation from the model trained with `supervised contrastive learning` clusters much more than the model trained with `cross-entropy`.", "1125445": "About the model performance, the models trained with `supervised contrastive learning` consistently performed better (higher LB score) than the models trained with `cross-entropy`, I ran out of TPU quota, but I am sure I could push it easily to `0.900` tweaking the parameters a little more.", "1125617": "very interesting", "1126216": "The approach is definetly interesting but with one caveat. Our data has mislabelled examples (noisy labels) and I am afraid that Supervised Contrastive loss is not robust to noisy labels. It will be interesting to see results with just plain SimCLR(self supervision).", "1126758": "Hi @atharvap329 , according to the paper, supervised contrastive learning performed better than cross-entropy on a benchmark that evaluate performance of different levels of corruption on imagenet, I am not sure the kind of corruption, but it might be worth it to git it a try.", "1126818": "They mention the corruption of images due to image images but its worth a try for sure it may happen that supervised contrastive loss may inherently be actually robust to noise(which will be a new finding :p). Looking forward to your results.", "1127635": "Interesting. Need to be looked up :)", "1128194": "Intersting. (Y)", "1171609": "For those interested in this method I just wrote [an article about it](https://medium.com/towards-artificial-intelligence/supervised-contrastive-learning-for-cassava-leaf-disease-classification-9dd47779a966), using this competition's data as a use case" }, "source": "meta" }