![\"\"](\"https://i.ibb.co/JvXgFKx/berlow.png\")
Self-supervised learning (SSL) is rapidly closing
\nthe gap with supervised methods on large computer vision benchmarks. A successful approach
\nto SSL is to learn representations which are invariant to distortions of the input sample. However, a
\nrecurring issue with this approach is the existence
\nof trivial constant representations. Most current
\nmethods avoid such collapsed solutions by careful
\nimplementation details. We propose an objective
\nfunction that naturally avoids such collapse by
\nmeasuring the cross-correlation matrix between
\nthe outputs of two identical networks fed with distorted versions of a sample, and making it as close
\nto the identity matrix as possible. This causes the
\nrepresentation vectors of distorted versions of a
\nsample to be similar, while minimizing the redundancy between the components of these vectors.
\nThe method is called BARLOW TWINS, owing to
\nneuroscientist H. Barlow’s redundancy-reduction
\nprinciple applied to a pair of identical networks.
\nBARLOW TWINS does not require large batches
\nnor asymmetry between the network twins such
\nas a predictor network, gradient stopping, or a
\nmoving average on the weight updates. It allows
\nthe use of very high-dimensional output vectors.
\nBARLOW TWINS outperforms previous methods
\non ImageNet for semi-supervised classification in
\nthe low-data regime, and is on par with current
\nstate of the art for ImageNet classification with
\na linear classifier head, and for transfer tasks of
\nclassification and object detection.
![\"\"](\"https://i.ibb.co/m8xY2Yj/berlo.png\")
paper : https://arxiv.org/pdf/2103.03230.pdf
\ngithub : https://github.com/facebookresearch/barlowtwins
", "messages": [ { "id": "1231206", "postDate": "03/08/2021 18:39:56", "content": "Self-supervised learning (SSL) is rapidly closing
\nthe gap with supervised methods on large computer vision benchmarks. A successful approach
\nto SSL is to learn representations which are invariant to distortions of the input sample. However, a
\nrecurring issue with this approach is the existence
\nof trivial constant representations. Most current
\nmethods avoid such collapsed solutions by careful
\nimplementation details. We propose an objective
\nfunction that naturally avoids such collapse by
\nmeasuring the cross-correlation matrix between
\nthe outputs of two identical networks fed with distorted versions of a sample, and making it as close
\nto the identity matrix as possible. This causes the
\nrepresentation vectors of distorted versions of a
\nsample to be similar, while minimizing the redundancy between the components of these vectors.
\nThe method is called BARLOW TWINS, owing to
\nneuroscientist H. Barlow’s redundancy-reduction
\nprinciple applied to a pair of identical networks.
\nBARLOW TWINS does not require large batches
\nnor asymmetry between the network twins such
\nas a predictor network, gradient stopping, or a
\nmoving average on the weight updates. It allows
\nthe use of very high-dimensional output vectors.
\nBARLOW TWINS outperforms previous methods
\non ImageNet for semi-supervised classification in
\nthe low-data regime, and is on par with current
\nstate of the art for ImageNet classification with
\na linear classifier head, and for transfer tasks of
\nclassification and object detection.
paper : https://arxiv.org/pdf/2103.03230.pdf
\ngithub : https://github.com/facebookresearch/barlowtwins
", "rawMarkdown": "# ABSTRACT \n\nSelf-supervised learning (SSL) is rapidly closing\nthe gap with supervised methods on large computer vision benchmarks. A successful approach\nto SSL is to learn representations which are invariant to distortions of the input sample. However, a\nrecurring issue with this approach is the existence\nof trivial constant representations. Most current\nmethods avoid such collapsed solutions by careful\nimplementation details. We propose an objective\nfunction that naturally avoids such collapse by\nmeasuring the cross-correlation matrix between\nthe outputs of two identical networks fed with distorted versions of a sample, and making it as close\nto the identity matrix as possible. This causes the\nrepresentation vectors of distorted versions of a\nsample to be similar, while minimizing the redundancy between the components of these vectors.\nThe method is called BARLOW TWINS, owing to\nneuroscientist H. Barlow’s redundancy-reduction\nprinciple applied to a pair of identical networks.\nBARLOW TWINS does not require large batches\nnor asymmetry between the network twins such\nas a predictor network, gradient stopping, or a\nmoving average on the weight updates. It allows\nthe use of very high-dimensional output vectors.\nBARLOW TWINS outperforms previous methods\non ImageNet for semi-supervised classification in\nthe low-data regime, and is on par with current\nstate of the art for ImageNet classification with\na linear classifier head, and for transfer tasks of\nclassification and object detection. \n\n\n\n\n\n\npaper : https://arxiv.org/pdf/2103.03230.pdf\n\ngithub : https://github.com/facebookresearch/barlowtwins", "votes": null }, { "id": "1231216", "postDate": "03/08/2021 18:45:48", "content": "for learning purpose you might want to read below 2 contents :
\nHigh-performance self-supervised image classification with contrastive clustering
Facebook’s New AI Teaches Itself to See With Less Human Help
Thank u for ur sharing!
", "rawMarkdown": "Thank u for ur sharing!", "votes": null }, { "id": "1231945", "postDate": "03/09/2021 11:49:03", "content": "I've been itching to use self-supervised for a few competitions now, but in reality, the batch sizes required to get it to work is usually unfeasible for us mere mortals
\n\n\nWe find that, unlike SIMCLR, our model is robust to small batch sizes (Fig. 2), with a performance almost unaffected for a batch as small as 256
\n
bruh, batch_size=4 is big for me right now 😭
", "rawMarkdown": "I've been itching to use self-supervised for a few competitions now, but in reality, the batch sizes required to get it to work is usually unfeasible for us mere mortals\n\n> We find that, unlike SIMCLR, our model is robust to small batch sizes (Fig. 2), with a performance almost unaffected for a batch as small as 256\n\nbruh, batch_size=4 is big for me right now 😭", "votes": null }, { "id": "1232463", "postDate": "03/09/2021 19:05:30", "content": "Thanks for sharing @mobassir !
\nI see you're getting close to discussion GM :)
hahahahah 😄
", "rawMarkdown": "hahahahah 😄", "votes": null }, { "id": "1234400", "postDate": "03/11/2021 08:15:53", "content": "this is interesting.
\nhow about plugging into my teach-student attention loss framework for consistency loss?
great idea
", "rawMarkdown": "great idea", "votes": null }, { "id": "1234717", "postDate": "03/11/2021 14:20:30", "content": "the problem with self-supervised learning (like contrastive loss) for this competition is the signal in the image is very small. We are detecting lines, more correctly the endpoint of lines. this is only about 5% of the total area of the image.
\nthe self-learning algorithm has to figure out that the target of interest area is that 5%.
\nyou can use visualization tools like CAM, etc to find out what the trained network thinks when it finds 2 images that are similar when using contrastive learning.
\nin order for contrastive learning to focus on that 5% area, your augmentation has to be cleverly designed. or your supervision (or rather self-supervision) has to be smart.
\nhence of the off shelf methods/code may not be directly applicable here.
\nIt does not mean they don't work. you would have to make the necessary modifications.
", "rawMarkdown": "the problem with self-supervised learning (like contrastive loss) for this competition is the signal in the image is very small. We are detecting lines, more correctly the endpoint of lines. this is only about 5% of the total area of the image.\n\nthe self-learning algorithm has to figure out that the target of interest area is that 5%. \n\nyou can use visualization tools like CAM, etc to find out what the trained network thinks when it finds 2 images that are similar when using contrastive learning.\n\nin order for contrastive learning to focus on that 5% area, your augmentation has to be cleverly designed. or your supervision (or rather self-supervision) has to be smart.\n\nhence of the off shelf methods/code may not be directly applicable here.\n\nIt does not mean they don't work. you would have to make the necessary modifications.", "votes": null }, { "id": "1241366", "postDate": "03/17/2021 02:36:03", "content": "@mobassir Thanks so much for sharing this impressive paper Mobassir . Insightful
", "rawMarkdown": "mobassir Thanks so much for sharing this impressive paper Mobassir . Insightful", "votes": null }, { "id": "1719326", "postDate": "03/11/2022 16:28:26", "content": "Hi @mobassir @anjum48 @pukkinming, I have created a list of videos covering all the topics related to self-supervised learning. I have tried to pull in state of art resources for the same. Hope it helps all. https://www.kaggle.com/getting-started/312197
", "rawMarkdown": "Hi @mobassir @anjum48 @pukkinming, I have created a list of videos covering all the topics related to self-supervised learning. I have tried to pull in state of art resources for the same. Hope it helps all. https://www.kaggle.com/getting-started/312197", "votes": null } ], "comments": [ { "id": 1231216, "author_name": "mobassir", "author_url": "", "post_date": "03/08/2021 18:45:48", "content": "for learning purpose you might want to read below 2 contents :
\nHigh-performance self-supervised image classification with contrastive clustering
Facebook’s New AI Teaches Itself to See With Less Human Help
Thank u for ur sharing!
", "votes": null, "replies": [] }, { "id": 1231945, "author_name": "anjum48", "author_url": "", "post_date": "03/09/2021 11:49:03", "content": "I've been itching to use self-supervised for a few competitions now, but in reality, the batch sizes required to get it to work is usually unfeasible for us mere mortals
\n\n\nWe find that, unlike SIMCLR, our model is robust to small batch sizes (Fig. 2), with a performance almost unaffected for a batch as small as 256
\n
bruh, batch_size=4 is big for me right now 😭
", "votes": null, "replies": [] }, { "id": 1232463, "author_name": "theoviel", "author_url": "", "post_date": "03/09/2021 19:05:30", "content": "Thanks for sharing @mobassir !
\nI see you're getting close to discussion GM :)
hahahahah 😄
", "votes": null, "replies": [] } ] }, { "id": 1234400, "author_name": "hengck23", "author_url": "", "post_date": "03/11/2021 08:15:53", "content": "this is interesting.
\nhow about plugging into my teach-student attention loss framework for consistency loss?
great idea
", "votes": null, "replies": [] } ] }, { "id": 1234717, "author_name": "hengck23", "author_url": "", "post_date": "03/11/2021 14:20:30", "content": "the problem with self-supervised learning (like contrastive loss) for this competition is the signal in the image is very small. We are detecting lines, more correctly the endpoint of lines. this is only about 5% of the total area of the image.
\nthe self-learning algorithm has to figure out that the target of interest area is that 5%.
\nyou can use visualization tools like CAM, etc to find out what the trained network thinks when it finds 2 images that are similar when using contrastive learning.
\nin order for contrastive learning to focus on that 5% area, your augmentation has to be cleverly designed. or your supervision (or rather self-supervision) has to be smart.
\nhence of the off shelf methods/code may not be directly applicable here.
\nIt does not mean they don't work. you would have to make the necessary modifications.
", "votes": null, "replies": [] }, { "id": 1241366, "author_name": "usharengaraju", "author_url": "", "post_date": "03/17/2021 02:36:03", "content": "@mobassir Thanks so much for sharing this impressive paper Mobassir . Insightful
", "votes": null, "replies": [] }, { "id": 1719326, "author_name": "shivkumarganesh", "author_url": "", "post_date": "03/11/2022 16:28:26", "content": "Hi @mobassir @anjum48 @pukkinming, I have created a list of videos covering all the topics related to self-supervised learning. I have tried to pull in state of art resources for the same. Hope it helps all. https://www.kaggle.com/getting-started/312197
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1231206": "# ABSTRACT \n\nSelf-supervised learning (SSL) is rapidly closing\nthe gap with supervised methods on large computer vision benchmarks. A successful approach\nto SSL is to learn representations which are invariant to distortions of the input sample. However, a\nrecurring issue with this approach is the existence\nof trivial constant representations. Most current\nmethods avoid such collapsed solutions by careful\nimplementation details. We propose an objective\nfunction that naturally avoids such collapse by\nmeasuring the cross-correlation matrix between\nthe outputs of two identical networks fed with distorted versions of a sample, and making it as close\nto the identity matrix as possible. This causes the\nrepresentation vectors of distorted versions of a\nsample to be similar, while minimizing the redundancy between the components of these vectors.\nThe method is called BARLOW TWINS, owing to\nneuroscientist H. Barlow’s redundancy-reduction\nprinciple applied to a pair of identical networks.\nBARLOW TWINS does not require large batches\nnor asymmetry between the network twins such\nas a predictor network, gradient stopping, or a\nmoving average on the weight updates. It allows\nthe use of very high-dimensional output vectors.\nBARLOW TWINS outperforms previous methods\non ImageNet for semi-supervised classification in\nthe low-data regime, and is on par with current\nstate of the art for ImageNet classification with\na linear classifier head, and for transfer tasks of\nclassification and object detection. \n\n\n\n\n\n\npaper : https://arxiv.org/pdf/2103.03230.pdf\n\ngithub : https://github.com/facebookresearch/barlowtwins", "1231216": "for learning purpose you might want to read below 2 contents : \n1. [High-performance self-supervised image classification with contrastive clustering](https://ai.facebook.com/blog/high-performance-self-supervised-image-classification-with-contrastive-clustering/)\n\n2. [ Facebook’s New AI Teaches Itself to See With Less Human Help](https://www.wired.com/story/facebook-new-ai-teaches-itself-see-less-human-help/?fbclid=IwAR31PPTTGLIr5XnuDEe7Shzy3lvp2xkL6Jyo3ZdY8tHtwzJiHjGxoeCg1UQ)", "1231402": "Thank u for ur sharing!", "1231945": "I've been itching to use self-supervised for a few competitions now, but in reality, the batch sizes required to get it to work is usually unfeasible for us mere mortals\n\n> We find that, unlike SIMCLR, our model is robust to small batch sizes (Fig. 2), with a performance almost unaffected for a batch as small as 256\n\nbruh, batch_size=4 is big for me right now 😭", "1232463": "Thanks for sharing @mobassir !\nI see you're getting close to discussion GM :)", "1232465": "hahahahah 😄", "1234400": "this is interesting.\nhow about plugging into my teach-student attention loss framework for consistency loss?", "1234493": "great idea", "1234717": "the problem with self-supervised learning (like contrastive loss) for this competition is the signal in the image is very small. We are detecting lines, more correctly the endpoint of lines. this is only about 5% of the total area of the image.\n\nthe self-learning algorithm has to figure out that the target of interest area is that 5%. \n\nyou can use visualization tools like CAM, etc to find out what the trained network thinks when it finds 2 images that are similar when using contrastive learning.\n\nin order for contrastive learning to focus on that 5% area, your augmentation has to be cleverly designed. or your supervision (or rather self-supervision) has to be smart.\n\nhence of the off shelf methods/code may not be directly applicable here.\n\nIt does not mean they don't work. you would have to make the necessary modifications.", "1241366": "mobassir Thanks so much for sharing this impressive paper Mobassir . Insightful", "1719326": "Hi @mobassir @anjum48 @pukkinming, I have created a list of videos covering all the topics related to self-supervised learning. I have tried to pull in state of art resources for the same. Hope it helps all. https://www.kaggle.com/getting-started/312197" }, "source": "meta" }