Did anybody try contrastive learning things like simsaim or something?
", "messages": [ { "id": 1144889, "postDate": "2021-01-08T18:21:12.513Z", "content": "Did anybody try contrastive learning things like simsaim or something?
", "rawMarkdown": "Did anybody try contrastive learning things like simsaim or something?", "votes": 2 }, { "id": 1144930, "postDate": "2021-01-08T18:49:54.083Z", "rawMarkdown": "", "votes": 1, "isDeleted": true, "replies": [ { "id": 1144984, "postDate": "2021-01-08T19:47:37.293Z", "content": "Ohh, I tried SIMSAIM, and yea it also didn't perform well.
\nContrastive learning methods are supposed to be good for noisy labels rit?
There are two things which you assume wrong.
\n1) Both SIMSAIM and SWAV are self supervised methods which were trained with a large batch size like 4092, thus ensuring variability of images in a single batch to generate better representations.
\n2) These are self supervised methods which will learn good representations but here may not be applicable since when you finetune the good representation version of model, you will still have noisy labels thus models wont learn to distunguish between noise.
The ideas are good but used in just different settings.
", "rawMarkdown": "There are two things which you assume wrong. \n1) Both SIMSAIM and SWAV are self supervised methods which were trained with a large batch size like 4092, thus ensuring variability of images in a single batch to generate better representations.\n2) These are self supervised methods which will learn good representations but here may not be applicable since when you finetune the good representation version of model, you will still have noisy labels thus models wont learn to distunguish between noise. \n\nThe ideas are good but used in just different settings.", "votes": 4 }, { "id": 1145811, "postDate": "2021-01-09T11:04:23.287Z", "content": "Well, I totally agree to your last point \"The model still won't learn to distinguish between noise\".
\nBut What I thought is wat if we are capable of learning perfect representations and then add a simple MLP on top of the bacbone to finetune on less data which are manually sampled(few shot learning may be).
\nI am not aware of SWAV, but SIMSAIM paper reported decent accuracy on ImageNet with batch size 64. Rest of the representation learning methods mostly are using large batch sizes as you told.
Again, the question arises how do you create a clean dataset :P.
", "rawMarkdown": "Again, the question arises how do you create a clean dataset :P. " }, { "id": 1175693, "postDate": "2021-01-29T09:37:32.673Z", "content": "Someone who is new to self-supervised learning, refer this :
\nhttps://ai.stackexchange.com/questions/10623/what-is-self-supervised-learning-in-machine-learning
I got introduced to this new method, after going through the above discussion.
", "rawMarkdown": "Someone who is **new to self-supervised learning**, refer this :\nhttps://ai.stackexchange.com/questions/10623/what-is-self-supervised-learning-in-machine-learning\n\nI got introduced to this new method, after going through the above discussion.", "votes": 1 }, { "id": 1175989, "postDate": "2021-01-29T12:42:28.963Z", "content": "Yes, self supervision learning is a way to cleverly use the unlabeled data to get a significant boost in the performance. Representation techniques like simsaim,simclr will learn compatible representations(if provided with good amount of augmentation) without even looking to their respective labels. I tried using SIMSAIM on our cassava data here and clustered the representations here.
\nI would love to listen your comments on these!
\nThanks!
Ohh, I tried SIMSAIM, and yea it also didn't perform well.
\nContrastive learning methods are supposed to be good for noisy labels rit?
There are two things which you assume wrong.
\n1) Both SIMSAIM and SWAV are self supervised methods which were trained with a large batch size like 4092, thus ensuring variability of images in a single batch to generate better representations.
\n2) These are self supervised methods which will learn good representations but here may not be applicable since when you finetune the good representation version of model, you will still have noisy labels thus models wont learn to distunguish between noise.
The ideas are good but used in just different settings.
", "votes": 4, "replies": [] }, { "id": 1145811, "author_name": "manoj akondi", "author_url": "", "post_date": "2021-01-09T11:04:23.287000", "content": "Well, I totally agree to your last point \"The model still won't learn to distinguish between noise\".
\nBut What I thought is wat if we are capable of learning perfect representations and then add a simple MLP on top of the bacbone to finetune on less data which are manually sampled(few shot learning may be).
\nI am not aware of SWAV, but SIMSAIM paper reported decent accuracy on ImageNet with batch size 64. Rest of the representation learning methods mostly are using large batch sizes as you told.
Again, the question arises how do you create a clean dataset :P.
", "votes": 0, "replies": [] }, { "id": 1175693, "author_name": "Kishan Joshi", "author_url": "", "post_date": "2021-01-29T09:37:32.673000", "content": "Someone who is new to self-supervised learning, refer this :
\nhttps://ai.stackexchange.com/questions/10623/what-is-self-supervised-learning-in-machine-learning
I got introduced to this new method, after going through the above discussion.
", "votes": 1, "replies": [] }, { "id": 1175989, "author_name": "manoj akondi", "author_url": "", "post_date": "2021-01-29T12:42:28.963000", "content": "Yes, self supervision learning is a way to cleverly use the unlabeled data to get a significant boost in the performance. Representation techniques like simsaim,simclr will learn compatible representations(if provided with good amount of augmentation) without even looking to their respective labels. I tried using SIMSAIM on our cassava data here and clustered the representations here.
\nI would love to listen your comments on these!
\nThanks!