![\"\"](\"https://raw.githubusercontent.com/clovaai/assembled-cnn/master/figures/summary_architecture.png\")
I have been reading few papers recently with a hope to apply best ideas to this competition; thought it would be nice to share.
\n\nRecent studies in image classification have demonstrated\na variety of techniques for improving the performance of\nConvolutional Neural Networks (CNNs). However, attempts to combine existing techniques to create a practical model are still uncommon. In this study, we carry\nout extensive experiments to validate that carefully assembling these techniques and applying them to a basic CNN\nmodel in combination can improve the accuracy and robustness of the model while minimizing the loss of throughput. For example, our proposed ResNet-50 shows an improvement in top-1 accuracy from 76.3% to 82.78%, and\nan mCE improvement from 76.0% to 48.9%, on the ImageNet ILSVRC2012 validation set. With these improvements, inference throughput only decreases from 536 to 312.\nThe resulting model significantly outperforms state-of-theart models with similar accuracy in terms of mCE and inference throughput. To verify the performance improvement\nin transfer learning, fine grained classification and image\nretrieval tasks were tested on several open datasets and\nshowed that the improvement to backbone network performance boosted transfer learning performance significantly.\nOur approach achieved 1st place in the iFood Competition\nFine-Grained Visual Recognition at CVPR 2019 1\n, and the\nsource code and trained models will be made publicly available
Overfitting & underfitting and stable training are an important challenges in machine learning. Current approaches for these issues are mixup, SamplePairing and BC learning. In our work, we state the hypothesis that mixing many images together can be more effective than just two. Batchboost pipeline has three stages: (a) pairing: method of selecting two samples. (b) mixing: how to create a new one from two samples. (c) feeding: combining mixed samples with new ones from dataset into batch (with ratio γ). Note that sample that appears in our batch propagates with subsequent iterations with less and less importance until the end of training. Pairing stage calculates the error per sample, sorts the samples and pairs with strategy: hardest with easiest one, than mixing stage merges two samples using mixup, x1+(1−λ)x2. Finally, feeding stage combines new samples with mixed by ratio 1:1. Batchboost has 0.5-3% better accuracy than the current state-of-the-art mixup regularization on CIFAR-10 & Fashion-MNIST. Our method is slightly better than SamplePairing technique on small datasets (up to 5%). Batchboost provides stable training on not tuned parameters (like weight decay), thus its a good method to test performance of different architectures
![\"\"](\"https://raw.githubusercontent.com/maciejczyzewski/batchboost/master/figures/figure-abstract.png\")
Adversarial examples are commonly viewed as a threat\nto ConvNets. Here we present an opposite perspective: adversarial examples can be used to improve image recognition models if harnessed in the right manner. We propose\nAdvProp, an enhanced adversarial training scheme which\ntreats adversarial examples as additional examples, to prevent overfitting. Key to our method is the usage of a separate auxiliary batch norm for adversarial examples, as they\nhave different underlying distributions to normal examples.\nWe show that AdvProp improves a wide range of models\non various image recognition tasks and performs better\nwhen the models are bigger. For instance, by applying\nAdvProp to the latest EfficientNet-B7 [28] on ImageNet, we\nachieve significant improvements on ImageNet (+0.7%),\nImageNet-C (+6.5%), ImageNet-A (+7.0%), StylizedImageNet (+4.8%). With an enhanced EfficientNet-B8,\nour method achieves the state-of-the-art 85.5% ImageNet\ntop-1 accuracy without extra data. This result even\nsurpasses the best model in [20] which is trained with\n3.5B Instagram images (∼3000× more than ImageNet)\nand ∼9.4× more parameters
![\"\"](\"https://deeplearn.org/arxiv_files/1911.09665v1/x3.png\")
\n", "messages": [ { "id": "729397", "postDate": "01/26/2020 06:14:50", "content": "I will be updating this list as I find more relevant paper; if you have any, please add to this list
\n
I have been reading few papers recently with a hope to apply best ideas to this competition; thought it would be nice to share.
\n\nRecent studies in image classification have demonstrated\na variety of techniques for improving the performance of\nConvolutional Neural Networks (CNNs). However, attempts to combine existing techniques to create a practical model are still uncommon. In this study, we carry\nout extensive experiments to validate that carefully assembling these techniques and applying them to a basic CNN\nmodel in combination can improve the accuracy and robustness of the model while minimizing the loss of throughput. For example, our proposed ResNet-50 shows an improvement in top-1 accuracy from 76.3% to 82.78%, and\nan mCE improvement from 76.0% to 48.9%, on the ImageNet ILSVRC2012 validation set. With these improvements, inference throughput only decreases from 536 to 312.\nThe resulting model significantly outperforms state-of-theart models with similar accuracy in terms of mCE and inference throughput. To verify the performance improvement\nin transfer learning, fine grained classification and image\nretrieval tasks were tested on several open datasets and\nshowed that the improvement to backbone network performance boosted transfer learning performance significantly.\nOur approach achieved 1st place in the iFood Competition\nFine-Grained Visual Recognition at CVPR 2019 1\n, and the\nsource code and trained models will be made publicly available
Overfitting & underfitting and stable training are an important challenges in machine learning. Current approaches for these issues are mixup, SamplePairing and BC learning. In our work, we state the hypothesis that mixing many images together can be more effective than just two. Batchboost pipeline has three stages: (a) pairing: method of selecting two samples. (b) mixing: how to create a new one from two samples. (c) feeding: combining mixed samples with new ones from dataset into batch (with ratio γ). Note that sample that appears in our batch propagates with subsequent iterations with less and less importance until the end of training. Pairing stage calculates the error per sample, sorts the samples and pairs with strategy: hardest with easiest one, than mixing stage merges two samples using mixup, x1+(1−λ)x2. Finally, feeding stage combines new samples with mixed by ratio 1:1. Batchboost has 0.5-3% better accuracy than the current state-of-the-art mixup regularization on CIFAR-10 & Fashion-MNIST. Our method is slightly better than SamplePairing technique on small datasets (up to 5%). Batchboost provides stable training on not tuned parameters (like weight decay), thus its a good method to test performance of different architectures
Adversarial examples are commonly viewed as a threat\nto ConvNets. Here we present an opposite perspective: adversarial examples can be used to improve image recognition models if harnessed in the right manner. We propose\nAdvProp, an enhanced adversarial training scheme which\ntreats adversarial examples as additional examples, to prevent overfitting. Key to our method is the usage of a separate auxiliary batch norm for adversarial examples, as they\nhave different underlying distributions to normal examples.\nWe show that AdvProp improves a wide range of models\non various image recognition tasks and performs better\nwhen the models are bigger. For instance, by applying\nAdvProp to the latest EfficientNet-B7 [28] on ImageNet, we\nachieve significant improvements on ImageNet (+0.7%),\nImageNet-C (+6.5%), ImageNet-A (+7.0%), StylizedImageNet (+4.8%). With an enhanced EfficientNet-B8,\nour method achieves the state-of-the-art 85.5% ImageNet\ntop-1 accuracy without extra data. This result even\nsurpasses the best model in [20] which is trained with\n3.5B Instagram images (∼3000× more than ImageNet)\nand ∼9.4× more parameters
\n", "rawMarkdown": "I have been reading few papers recently with a hope to apply best ideas to this competition; thought it would be nice to share.\n\n##1. [Compounding the Performance Improvements of Assembled Techniques in a Convolutional Neural Network](https://github.com/clovaai/assembled-cnn)\n`Recent studies in image classification have demonstrated\na variety of techniques for improving the performance of\nConvolutional Neural Networks (CNNs). However, attempts to combine existing techniques to create a practical model are still uncommon. In this study, we carry\nout extensive experiments to validate that carefully assembling these techniques and applying them to a basic CNN\nmodel in combination can improve the accuracy and robustness of the model while minimizing the loss of throughput. For example, our proposed ResNet-50 shows an improvement in top-1 accuracy from 76.3% to 82.78%, and\nan mCE improvement from 76.0% to 48.9%, on the ImageNet ILSVRC2012 validation set. With these improvements, inference throughput only decreases from 536 to 312.\nThe resulting model significantly outperforms state-of-theart models with similar accuracy in terms of mCE and inference throughput. To verify the performance improvement\nin transfer learning, fine grained classification and image\nretrieval tasks were tested on several open datasets and\nshowed that the improvement to backbone network performance boosted transfer learning performance significantly.\nOur approach achieved 1st place in the iFood Competition\nFine-Grained Visual Recognition at CVPR 2019 1\n, and the\nsource code and trained models will be made publicly available `\n\n\n\n##2. [batchboost: regularization for stabilizing training with resistance to underfitting & overfitting](https://github.com/maciejczyzewski/batchboost)\n`Overfitting & underfitting and stable training are an important challenges in machine learning. Current approaches for these issues are mixup, SamplePairing and BC learning. In our work, we state the hypothesis that mixing many images together can be more effective than just two. Batchboost pipeline has three stages: (a) pairing: method of selecting two samples. (b) mixing: how to create a new one from two samples. (c) feeding: combining mixed samples with new ones from dataset into batch (with ratio γ). Note that sample that appears in our batch propagates with subsequent iterations with less and less importance until the end of training. Pairing stage calculates the error per sample, sorts the samples and pairs with strategy: hardest with easiest one, than mixing stage merges two samples using mixup, x1+(1−λ)x2. Finally, feeding stage combines new samples with mixed by ratio 1:1. Batchboost has 0.5-3% better accuracy than the current state-of-the-art mixup regularization on CIFAR-10 & Fashion-MNIST. Our method is slightly better than SamplePairing technique on small datasets (up to 5%). Batchboost provides stable training on not tuned parameters (like weight decay), thus its a good method to test performance of different architectures`\n\n\n\n##3. [Adversarial Examples Improve Image Recognition](https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/split_batchnorm.py)\n`Adversarial examples are commonly viewed as a threat\nto ConvNets. Here we present an opposite perspective: adversarial examples can be used to improve image recognition models if harnessed in the right manner. We propose\nAdvProp, an enhanced adversarial training scheme which\ntreats adversarial examples as additional examples, to prevent overfitting. Key to our method is the usage of a separate auxiliary batch norm for adversarial examples, as they\nhave different underlying distributions to normal examples.\nWe show that AdvProp improves a wide range of models\non various image recognition tasks and performs better\nwhen the models are bigger. For instance, by applying\nAdvProp to the latest EfficientNet-B7 [28] on ImageNet, we\nachieve significant improvements on ImageNet (+0.7%),\nImageNet-C (+6.5%), ImageNet-A (+7.0%), StylizedImageNet (+4.8%). With an enhanced EfficientNet-B8,\nour method achieves the state-of-the-art 85.5% ImageNet\ntop-1 accuracy without extra data. This result even\nsurpasses the best model in [20] which is trained with\n3.5B Instagram images (∼3000× more than ImageNet)\nand ∼9.4× more parameters`\n\n\n\n\n\n> I will be updating this list as I find more relevant paper; if you have any, please add to this list", "votes": null }, { "id": "729550", "postDate": "01/26/2020 11:10:30", "content": "I will be updating this list as I find more relevant paper; if you have any, please add to this list
\n
FixMatch: Simplifying Semi-Supervised Learning with\nConsistency and Confidence\n\nSemi-supervised learning (SSL) provides an effective means of leveraging unlabeled data to improve a model’s performance. In this paper, we demonstrate the power of a simple combination of two common SSL methods: consistency regularization and pseudo-labeling. Our algorithm, FixMatch, first generates pseudo-labels using the model’s predictions on weakly-augmented unlabeled images. For a given image, the pseudo-label is only retained if the model produces a high-confidence prediction. The model is then trained to predict the pseudo-label when fed a stronglyaugmented version of the same image. Despite its simplicity, we show that FixMatch achieves state-of-the-art performance across a variety of standard semi-supervised learning benchmarks, including 94.93% accuracy on CIFAR-10 with 250 labels and 88.61% accuracy with 40 – just 4 labels per class. Since FixMatch bears many similarities to existing SSL methods that achieve worse performance, we carry out an extensive ablation study to tease apart the experimental factors that are most important to FixMatch’s success. We make our code available at https://github.com/google-research/fixmatch.\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F4033611%2F6b879944e6070b3867b1f1ffee99b29d%2FPL.png?generation=1580036801177014&alt=media\")
Self-training with Noisy Student improves ImageNet classification
\n\n```\nWe present a simple self-training method that achieves 88.4% top-1 accuracy on ImageNet, which is 2.0% better than the state-of-the-art model that requires 3.5B weakly labeled Instagram images. On robustness test sets, it improves ImageNet-A top-1 accuracy from 61.0% to 83.7%, reduces ImageNet-C mean corruption error from 45.7 to 28.3, and reduces ImageNet-P mean flip rate from 27.8 to 12.2.
\n\nTo achieve this result, we first train an EfficientNet model on labeled ImageNet images and use it as a teacher to generate pseudo labels on 300M unlabeled images. We then train a larger EfficientNet as a student model on the combination of labeled and pseudo labeled images. We iterate this process by putting back the student as the teacher. During the generation of the pseudo labels, the teacher is not noised so that the pseudo labels are as accurate as possible. However, during the learning of the student, we inject noise such as dropout, stochastic depth and data augmentation via RandAugment to the student so that the student generalizes better than the teacher\n```
", "rawMarkdown": "[Self-training with Noisy Student improves ImageNet classification](https://arxiv.org/abs/1911.04252)\n\n```\nWe present a simple self-training method that achieves 88.4% top-1 accuracy on ImageNet, which is 2.0% better than the state-of-the-art model that requires 3.5B weakly labeled Instagram images. On robustness test sets, it improves ImageNet-A top-1 accuracy from 61.0% to 83.7%, reduces ImageNet-C mean corruption error from 45.7 to 28.3, and reduces ImageNet-P mean flip rate from 27.8 to 12.2.\n\nTo achieve this result, we first train an EfficientNet model on labeled ImageNet images and use it as a teacher to generate pseudo labels on 300M unlabeled images. We then train a larger EfficientNet as a student model on the combination of labeled and pseudo labeled images. We iterate this process by putting back the student as the teacher. During the generation of the pseudo labels, the teacher is not noised so that the pseudo labels are as accurate as possible. However, during the learning of the student, we inject noise such as dropout, stochastic depth and data augmentation via RandAugment to the student so that the student generalizes better than the teacher\n```", "votes": null } ], "comments": [ { "id": 729550, "author_name": "andreyzotov", "author_url": "", "post_date": "01/26/2020 11:10:30", "content": "FixMatch: Simplifying Semi-Supervised Learning with\nConsistency and Confidence\n\nSemi-supervised learning (SSL) provides an effective means of leveraging unlabeled data to improve a model’s performance. In this paper, we demonstrate the power of a simple combination of two common SSL methods: consistency regularization and pseudo-labeling. Our algorithm, FixMatch, first generates pseudo-labels using the model’s predictions on weakly-augmented unlabeled images. For a given image, the pseudo-label is only retained if the model produces a high-confidence prediction. The model is then trained to predict the pseudo-label when fed a stronglyaugmented version of the same image. Despite its simplicity, we show that FixMatch achieves state-of-the-art performance across a variety of standard semi-supervised learning benchmarks, including 94.93% accuracy on CIFAR-10 with 250 labels and 88.61% accuracy with 40 – just 4 labels per class. Since FixMatch bears many similarities to existing SSL methods that achieve worse performance, we carry out an extensive ablation study to tease apart the experimental factors that are most important to FixMatch’s success. We make our code available at https://github.com/google-research/fixmatch.\n\n
Self-training with Noisy Student improves ImageNet classification
\n\n```\nWe present a simple self-training method that achieves 88.4% top-1 accuracy on ImageNet, which is 2.0% better than the state-of-the-art model that requires 3.5B weakly labeled Instagram images. On robustness test sets, it improves ImageNet-A top-1 accuracy from 61.0% to 83.7%, reduces ImageNet-C mean corruption error from 45.7 to 28.3, and reduces ImageNet-P mean flip rate from 27.8 to 12.2.
\n\nTo achieve this result, we first train an EfficientNet model on labeled ImageNet images and use it as a teacher to generate pseudo labels on 300M unlabeled images. We then train a larger EfficientNet as a student model on the combination of labeled and pseudo labeled images. We iterate this process by putting back the student as the teacher. During the generation of the pseudo labels, the teacher is not noised so that the pseudo labels are as accurate as possible. However, during the learning of the student, we inject noise such as dropout, stochastic depth and data augmentation via RandAugment to the student so that the student generalizes better than the teacher\n```
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "729397": "I have been reading few papers recently with a hope to apply best ideas to this competition; thought it would be nice to share.\n\n##1. [Compounding the Performance Improvements of Assembled Techniques in a Convolutional Neural Network](https://github.com/clovaai/assembled-cnn)\n`Recent studies in image classification have demonstrated\na variety of techniques for improving the performance of\nConvolutional Neural Networks (CNNs). However, attempts to combine existing techniques to create a practical model are still uncommon. In this study, we carry\nout extensive experiments to validate that carefully assembling these techniques and applying them to a basic CNN\nmodel in combination can improve the accuracy and robustness of the model while minimizing the loss of throughput. For example, our proposed ResNet-50 shows an improvement in top-1 accuracy from 76.3% to 82.78%, and\nan mCE improvement from 76.0% to 48.9%, on the ImageNet ILSVRC2012 validation set. With these improvements, inference throughput only decreases from 536 to 312.\nThe resulting model significantly outperforms state-of-theart models with similar accuracy in terms of mCE and inference throughput. To verify the performance improvement\nin transfer learning, fine grained classification and image\nretrieval tasks were tested on several open datasets and\nshowed that the improvement to backbone network performance boosted transfer learning performance significantly.\nOur approach achieved 1st place in the iFood Competition\nFine-Grained Visual Recognition at CVPR 2019 1\n, and the\nsource code and trained models will be made publicly available `\n\n\n\n##2. [batchboost: regularization for stabilizing training with resistance to underfitting & overfitting](https://github.com/maciejczyzewski/batchboost)\n`Overfitting & underfitting and stable training are an important challenges in machine learning. Current approaches for these issues are mixup, SamplePairing and BC learning. In our work, we state the hypothesis that mixing many images together can be more effective than just two. Batchboost pipeline has three stages: (a) pairing: method of selecting two samples. (b) mixing: how to create a new one from two samples. (c) feeding: combining mixed samples with new ones from dataset into batch (with ratio γ). Note that sample that appears in our batch propagates with subsequent iterations with less and less importance until the end of training. Pairing stage calculates the error per sample, sorts the samples and pairs with strategy: hardest with easiest one, than mixing stage merges two samples using mixup, x1+(1−λ)x2. Finally, feeding stage combines new samples with mixed by ratio 1:1. Batchboost has 0.5-3% better accuracy than the current state-of-the-art mixup regularization on CIFAR-10 & Fashion-MNIST. Our method is slightly better than SamplePairing technique on small datasets (up to 5%). Batchboost provides stable training on not tuned parameters (like weight decay), thus its a good method to test performance of different architectures`\n\n\n\n##3. [Adversarial Examples Improve Image Recognition](https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/split_batchnorm.py)\n`Adversarial examples are commonly viewed as a threat\nto ConvNets. Here we present an opposite perspective: adversarial examples can be used to improve image recognition models if harnessed in the right manner. We propose\nAdvProp, an enhanced adversarial training scheme which\ntreats adversarial examples as additional examples, to prevent overfitting. Key to our method is the usage of a separate auxiliary batch norm for adversarial examples, as they\nhave different underlying distributions to normal examples.\nWe show that AdvProp improves a wide range of models\non various image recognition tasks and performs better\nwhen the models are bigger. For instance, by applying\nAdvProp to the latest EfficientNet-B7 [28] on ImageNet, we\nachieve significant improvements on ImageNet (+0.7%),\nImageNet-C (+6.5%), ImageNet-A (+7.0%), StylizedImageNet (+4.8%). With an enhanced EfficientNet-B8,\nour method achieves the state-of-the-art 85.5% ImageNet\ntop-1 accuracy without extra data. This result even\nsurpasses the best model in [20] which is trained with\n3.5B Instagram images (∼3000× more than ImageNet)\nand ∼9.4× more parameters`\n\n\n\n\n\n> I will be updating this list as I find more relevant paper; if you have any, please add to this list", "729550": "[**FixMatch: Simplifying Semi-Supervised Learning with\nConsistency and Confidence**](https://arxiv.org/pdf/2001.07685.pdf)\n```\nSemi-supervised learning (SSL) provides an effective means of leveraging unlabeled data to improve a model’s performance. In this paper, we demonstrate the power of a simple combination of two common SSL methods: consistency regularization and pseudo-labeling. Our algorithm, FixMatch, first generates pseudo-labels using the model’s predictions on weakly-augmented unlabeled images. For a given image, the pseudo-label is only retained if the model produces a high-confidence prediction. The model is then trained to predict the pseudo-label when fed a stronglyaugmented version of the same image. Despite its simplicity, we show that FixMatch achieves state-of-the-art performance across a variety of standard semi-supervised learning benchmarks, including 94.93% accuracy on CIFAR-10 with 250 labels and 88.61% accuracy with 40 – just 4 labels per class. Since FixMatch bears many similarities to existing SSL methods that achieve worse performance, we carry out an extensive ablation study to tease apart the experimental factors that are most important to FixMatch’s success. We make our code available at https://github.com/google-research/fixmatch.\n```\n", "735259": "[Self-training with Noisy Student improves ImageNet classification](https://arxiv.org/abs/1911.04252)\n\n```\nWe present a simple self-training method that achieves 88.4% top-1 accuracy on ImageNet, which is 2.0% better than the state-of-the-art model that requires 3.5B weakly labeled Instagram images. On robustness test sets, it improves ImageNet-A top-1 accuracy from 61.0% to 83.7%, reduces ImageNet-C mean corruption error from 45.7 to 28.3, and reduces ImageNet-P mean flip rate from 27.8 to 12.2.\n\nTo achieve this result, we first train an EfficientNet model on labeled ImageNet images and use it as a teacher to generate pseudo labels on 300M unlabeled images. We then train a larger EfficientNet as a student model on the combination of labeled and pseudo labeled images. We iterate this process by putting back the student as the teacher. During the generation of the pseudo labels, the teacher is not noised so that the pseudo labels are as accurate as possible. However, during the learning of the student, we inject noise such as dropout, stochastic depth and data augmentation via RandAugment to the student so that the student generalizes better than the teacher\n```" }, "source": "meta" }