![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F19390%2F3fdbb6b475cd1c5efa06572460c2b783%2Fsegment-charts.jpg?generation=1572187112654283&alt=media\")
Thanks to all organizers and Kaggle for such an interesting dataset and competition, and congrats to everyone who finished 🎉
\n\nBelow is an overview of my solution. Code is at https://github.com/lopuhin/kaggle-kuzushiji-2019, model weights for the best model are at https://github.com/lopuhin/kaggle-kuzushiji-2019/releases/tag/v1.0
\n\nGeneral approach is as follows:
\n\nresnet152 backbone pretrained on ImageNet). Out-of-fold predictions are obtained for all 5 folds.seg_fp (segmentation false-positive) is added for bounding boxes which have low overlap with ground truth boxes, so classification model can correct errors of segmentation model. Classification model is trained on all folds. Models with resnet152 and resnext101_32x8d_wsl backbones are used, they are trained on large crops containing multiple symbols, using FPN and roi align with a classification head.Why such approach was chosen? There are two other candidate approaches:
\n\nNext come more details on each stage.
\n\nSegmentation into characters is done with a Faster-RCNN model with resnet152 backbone trained with torchvision. Only one class is used, so it does not try to predict the character class. This model trains very fast and gives high quality boxes. Competition F1 metric (assuming\nperfect prediction for the classes) was around ~0.99 on validation.
Some details:
\n\nresnet152 backbone worked a bit better than default resnet50 (even though it was not pre-trained on COCO, doing this would offer another small boost),Overall many more improvements are possible here: using mmdetection, better models, pre-training on COCO, blending predictions from different folds for submission, TTA, separate model to discard out-of-page symbols, etc. Still it seemed that classification was more important.
\n\nThis is implemented in https://github.com/lopuhin/kaggle-kuzushiji-2019/tree/master/kuzushiji/segment (which is based on reference torchvision detection code), dataset is defined in https://github.com/lopuhin/kaggle-kuzushiji-2019/blob/master/kuzushiji/segment/dataset.py
\n\nHere are validation F1 scores (assuming perfect class prediction) for resnet50 pretrained on COCO (orange), resnet152 pretrained on ImageNet (green), and same but with a better training schedule (blue, used for the submission).
\n\n
Classification is performed by a model which gets as input a large crop from the image (512x768 from 2500-3000px high image) which contains multiple characters. It also receives as input boudning boxes predicted by segmentation model (these are out-of-fold predictions). This is similar to multi-class detection, but with frozen bounding boxes. ResNet base is used, layer4 is discarded, features are extracted for each bounding box with roi_align from layer2 and layer3 and concatenated, and then passed into a classification head.
Some details:
\n\nresnext101_32x8d_wsl from https://github.com/facebookresearch/WSL-Imagesresnext101_32x8d_wsl on 2080ti, mixed precision training was required along with freezing first convolution and whole layer1 (as I learned from Arthur Kuzin who did quite well in OpenImages, this is a trick used in mmdetection: https://github.com/open-mmlab/mmdetection/blob/6668bf0368b7ec6e88bc01aebdc281d2f79ef0cb/mmdet/models/backbones/resnet.py#L460)resnext101_32x8d_wsl took around 15 hours to train on one 2080ti.Best single model without pseudolabelling obtained public LB score of 0.935, although score varied quite a lot between folds, most folds were in 0.925 - 0.930 range. A blend of resnet152 and resnext101_32x8d_wsl models across all folds scored 0.941 on the public LB.
Overall, many improvement are possible here, from just using bigger models and freezing less layers, to more work on training schedule, augmentations, etc.
\n\nSee https://github.com/lopuhin/kaggle-kuzushiji-2019/blob/master/kuzushiji/classify/main.py for the training script, https://github.com/lopuhin/kaggle-kuzushiji-2019/blob/master/kuzushiji/classify/models.py for the models, and https://github.com/lopuhin/kaggle-kuzushiji-2019/blob/master/kuzushiji/classify/dataset.py for the dataset and augmentations.
\n\nHere are validation F1 scores of several classification models (including psedulabeling, see below). resnet50 is orange, resnet152 is blue, resnext101_32x8d_wsl is violet, the same fine-tuned on pseduo-labels is red, and the same trained from scratch on pseduo-labels is green.
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F19390%2Ff06ac978504dcb33220f403a9b261ca8%2Fclassify-charts.jpg?generation=1572187271578401&alt=media\")
Pseudolabelling is a technique where we take confident predictions of our model on test data, and add this to train dataset. Even though the model is already confident in such predictions, they are still useful and improve quality, because they allow the model to adapt better to different domain, as each book has it's own character and paper style, each author has different writing, etc.
\n\nHere the simplest approach was chosen: most confident predictions were used for all test set, instead of splitting it by book. Top 80% most confident predictions from the blend were used, having accuracy >99% according to validation. Next, two kinds of models were trained (all based on resnext101_32x8d_wsl):
In both cases, models used both train and test data for training. Best single fine-tuned model scored 0.939 on the public LB. Best single from-scratch model scored 0.944 on the public LB (0.945 on private LB).
\n\nSee https://github.com/lopuhin/kaggle-kuzushiji-2019/blob/master/kuzushiji/classify/pseudolabel.py for a script which filters confident prediction. This data is added in the regular classification train script.
\n\nA simple blend worked already quite well, giving 0.943 public LB (without pseudolabelled from-scratch models). Adjusting coefficients of the models didn't improve the validation score, even though resnext101_32x8d_wsl models were noticeably better.
Since all models were trained across all folds, it was possible to train a second level model, a blend of LightGBM and XGBoost. This model was inspired by Pavel Ostyakov's solution to Cdiscount’s Image Classification Challenge, which was a classification problem with 5k classes:\nhttps://www.kaggle.com/c/cdiscount-image-classification-challenge/discussion/45733
\n\nEach of 4 model kinds from classification contributed classes and scores of top-3 predictions as features. Also max overlap with other bboxes was added. Then for each of all classes in top-3 predictions, and for a seg_fp class, we created one row with an extra feature candidate, which had a class as a value, and the target is binary: whether this candidate class was a true class which should be predicted. Then for each top-3 class, we added an extra binary feature which tells whether this class is a candidate class.
Here is a simplified example with 1 model and top-2 predictions, all rows created for one character prediction (seg_fp was encoded as -1, top0_s means top0_score, top0_is_c means top0_is_candidate)::
top0_cls top1_cls top0_s top1_s candidate top0_is_c top1_is_c y\n83 258 15.202 7.1246 83 True False True\n83 258 15.202 7.1246 258 False True False\n83 258 15.202 7.1246 -1 False False False\nXGBoost and LighGBM models were trained across all folds, and then blended. It was better to first apply models to fold predictions on test and then blend them.
\n\nSuch blend gives 0.949 on public LB.
\n\nI'm extremely bad at tuning such models, so there may be more improvements possible. Adjusting seg_fp ratio was tried and provided some boost on validation but didn't work on public LB.
Second level features are defined in https://github.com/lopuhin/kaggle-kuzushiji-2019/blob/master/kuzushiji/classify/level2_features.py and the models are built in https://github.com/lopuhin/kaggle-kuzushiji-2019/blob/master/kuzushiji/classify/level2.py
\n\nScores on public LB (private LB scores are very well correlated):
\n\nresnet50, fold0: 0.916resnet152, fold0: 0.926resnet152, fold4: 0.932 (same model as above, best fold)resnext101_32x8d_wsl, fold4: 0.934resnext101_32x8d_wsl, fold4, fine-tuned with pseudo-labels: 0.939resnext101_32x8d_wsl, fold4, re-trained with pseudo-labels: 0.944See https://github.com/lopuhin/kaggle-kuzushiji-2019#install and https://github.com/lopuhin/kaggle-kuzushiji-2019#run and also the script which in theory contains all steps https://github.com/lopuhin/kaggle-kuzushiji-2019/blob/master/run-all.sh but was not run in practice.
\n\nAlmost all models were trained on my home server with one 2080ti. resnet152 classification models were trained on GCP with P100 GPUs as they required 16 GB of memory and I had some GCP credits. A few models towards the end were trained on vast.ai.
All models are written with pytorch, detection models are based on torchvision. Apex is used for mixed precision training, and Albumentations for augmentations.
", "messages": [ { "id": 649071, "postDate": "2019-10-14T22:55:12.150Z", "content": "Thanks to all organizers and Kaggle for such an interesting dataset and competition, and congrats to everyone who finished 🎉
\n\nBelow is an overview of my solution. Code is at https://github.com/lopuhin/kaggle-kuzushiji-2019, model weights for the best model are at https://github.com/lopuhin/kaggle-kuzushiji-2019/releases/tag/v1.0
\n\nGeneral approach is as follows:
\n\nresnet152 backbone pretrained on ImageNet). Out-of-fold predictions are obtained for all 5 folds.seg_fp (segmentation false-positive) is added for bounding boxes which have low overlap with ground truth boxes, so classification model can correct errors of segmentation model. Classification model is trained on all folds. Models with resnet152 and resnext101_32x8d_wsl backbones are used, they are trained on large crops containing multiple symbols, using FPN and roi align with a classification head.Why such approach was chosen? There are two other candidate approaches:
\n\nNext come more details on each stage.
\n\nSegmentation into characters is done with a Faster-RCNN model with resnet152 backbone trained with torchvision. Only one class is used, so it does not try to predict the character class. This model trains very fast and gives high quality boxes. Competition F1 metric (assuming\nperfect prediction for the classes) was around ~0.99 on validation.
Some details:
\n\nresnet152 backbone worked a bit better than default resnet50 (even though it was not pre-trained on COCO, doing this would offer another small boost),Overall many more improvements are possible here: using mmdetection, better models, pre-training on COCO, blending predictions from different folds for submission, TTA, separate model to discard out-of-page symbols, etc. Still it seemed that classification was more important.
\n\nThis is implemented in https://github.com/lopuhin/kaggle-kuzushiji-2019/tree/master/kuzushiji/segment (which is based on reference torchvision detection code), dataset is defined in https://github.com/lopuhin/kaggle-kuzushiji-2019/blob/master/kuzushiji/segment/dataset.py
\n\nHere are validation F1 scores (assuming perfect class prediction) for resnet50 pretrained on COCO (orange), resnet152 pretrained on ImageNet (green), and same but with a better training schedule (blue, used for the submission).
\n\n
Classification is performed by a model which gets as input a large crop from the image (512x768 from 2500-3000px high image) which contains multiple characters. It also receives as input boudning boxes predicted by segmentation model (these are out-of-fold predictions). This is similar to multi-class detection, but with frozen bounding boxes. ResNet base is used, layer4 is discarded, features are extracted for each bounding box with roi_align from layer2 and layer3 and concatenated, and then passed into a classification head.
Some details:
\n\nresnext101_32x8d_wsl from https://github.com/facebookresearch/WSL-Imagesresnext101_32x8d_wsl on 2080ti, mixed precision training was required along with freezing first convolution and whole layer1 (as I learned from Arthur Kuzin who did quite well in OpenImages, this is a trick used in mmdetection: https://github.com/open-mmlab/mmdetection/blob/6668bf0368b7ec6e88bc01aebdc281d2f79ef0cb/mmdet/models/backbones/resnet.py#L460)resnext101_32x8d_wsl took around 15 hours to train on one 2080ti.Best single model without pseudolabelling obtained public LB score of 0.935, although score varied quite a lot between folds, most folds were in 0.925 - 0.930 range. A blend of resnet152 and resnext101_32x8d_wsl models across all folds scored 0.941 on the public LB.
Overall, many improvement are possible here, from just using bigger models and freezing less layers, to more work on training schedule, augmentations, etc.
\n\nSee https://github.com/lopuhin/kaggle-kuzushiji-2019/blob/master/kuzushiji/classify/main.py for the training script, https://github.com/lopuhin/kaggle-kuzushiji-2019/blob/master/kuzushiji/classify/models.py for the models, and https://github.com/lopuhin/kaggle-kuzushiji-2019/blob/master/kuzushiji/classify/dataset.py for the dataset and augmentations.
\n\nHere are validation F1 scores of several classification models (including psedulabeling, see below). resnet50 is orange, resnet152 is blue, resnext101_32x8d_wsl is violet, the same fine-tuned on pseduo-labels is red, and the same trained from scratch on pseduo-labels is green.
Pseudolabelling is a technique where we take confident predictions of our model on test data, and add this to train dataset. Even though the model is already confident in such predictions, they are still useful and improve quality, because they allow the model to adapt better to different domain, as each book has it's own character and paper style, each author has different writing, etc.
\n\nHere the simplest approach was chosen: most confident predictions were used for all test set, instead of splitting it by book. Top 80% most confident predictions from the blend were used, having accuracy >99% according to validation. Next, two kinds of models were trained (all based on resnext101_32x8d_wsl):
In both cases, models used both train and test data for training. Best single fine-tuned model scored 0.939 on the public LB. Best single from-scratch model scored 0.944 on the public LB (0.945 on private LB).
\n\nSee https://github.com/lopuhin/kaggle-kuzushiji-2019/blob/master/kuzushiji/classify/pseudolabel.py for a script which filters confident prediction. This data is added in the regular classification train script.
\n\nA simple blend worked already quite well, giving 0.943 public LB (without pseudolabelled from-scratch models). Adjusting coefficients of the models didn't improve the validation score, even though resnext101_32x8d_wsl models were noticeably better.
Since all models were trained across all folds, it was possible to train a second level model, a blend of LightGBM and XGBoost. This model was inspired by Pavel Ostyakov's solution to Cdiscount’s Image Classification Challenge, which was a classification problem with 5k classes:\nhttps://www.kaggle.com/c/cdiscount-image-classification-challenge/discussion/45733
\n\nEach of 4 model kinds from classification contributed classes and scores of top-3 predictions as features. Also max overlap with other bboxes was added. Then for each of all classes in top-3 predictions, and for a seg_fp class, we created one row with an extra feature candidate, which had a class as a value, and the target is binary: whether this candidate class was a true class which should be predicted. Then for each top-3 class, we added an extra binary feature which tells whether this class is a candidate class.
Here is a simplified example with 1 model and top-2 predictions, all rows created for one character prediction (seg_fp was encoded as -1, top0_s means top0_score, top0_is_c means top0_is_candidate)::
top0_cls top1_cls top0_s top1_s candidate top0_is_c top1_is_c y\n83 258 15.202 7.1246 83 True False True\n83 258 15.202 7.1246 258 False True False\n83 258 15.202 7.1246 -1 False False False\nXGBoost and LighGBM models were trained across all folds, and then blended. It was better to first apply models to fold predictions on test and then blend them.
\n\nSuch blend gives 0.949 on public LB.
\n\nI'm extremely bad at tuning such models, so there may be more improvements possible. Adjusting seg_fp ratio was tried and provided some boost on validation but didn't work on public LB.
Second level features are defined in https://github.com/lopuhin/kaggle-kuzushiji-2019/blob/master/kuzushiji/classify/level2_features.py and the models are built in https://github.com/lopuhin/kaggle-kuzushiji-2019/blob/master/kuzushiji/classify/level2.py
\n\nScores on public LB (private LB scores are very well correlated):
\n\nresnet50, fold0: 0.916resnet152, fold0: 0.926resnet152, fold4: 0.932 (same model as above, best fold)resnext101_32x8d_wsl, fold4: 0.934resnext101_32x8d_wsl, fold4, fine-tuned with pseudo-labels: 0.939resnext101_32x8d_wsl, fold4, re-trained with pseudo-labels: 0.944See https://github.com/lopuhin/kaggle-kuzushiji-2019#install and https://github.com/lopuhin/kaggle-kuzushiji-2019#run and also the script which in theory contains all steps https://github.com/lopuhin/kaggle-kuzushiji-2019/blob/master/run-all.sh but was not run in practice.
\n\nAlmost all models were trained on my home server with one 2080ti. resnet152 classification models were trained on GCP with P100 GPUs as they required 16 GB of memory and I had some GCP credits. A few models towards the end were trained on vast.ai.
All models are written with pytorch, detection models are based on torchvision. Apex is used for mixed precision training, and Albumentations for augmentations.
", "rawMarkdown": "Thanks to all organizers and Kaggle for such an interesting dataset and competition, and congrats to everyone who finished 🎉 \n\nBelow is an overview of my solution. Code is at https://github.com/lopuhin/kaggle-kuzushiji-2019, model weights for the best model are at https://github.com/lopuhin/kaggle-kuzushiji-2019/releases/tag/v1.0\n\nGeneral approach is as follows:\n\n- Dataset is split into 5 folds by book.\n- Class-agnostic bounding boxes are predicted for all characters using an object detection network (with ``resnet152`` backbone pretrained on ImageNet). Out-of-fold predictions are obtained for all 5 folds.\n- A \"classification\" model is trained using OOF detection predictions. An extra class ``seg_fp`` (segmentation false-positive) is added for bounding boxes which have low overlap with ground truth boxes, so classification model can correct errors of segmentation model. Classification model is trained on all folds. Models with ``resnet152`` and ``resnext101_32x8d_wsl`` backbones are used, they are trained on large crops containing multiple symbols, using FPN and roi align with a classification head.\n- Pseudolabelling is performed, in OCR terms this is similar to \"writer adaptation\", although here it is applied to the whole test for simplicity.\n- A second level model is trained on classification predictions, which creates the final submission.\n\nWhy such approach was chosen? There are two other candidate approaches:\n\n- End-to-end model which does detection and classification (e.g. Faster-RCNN). This may be possible with some effort, but here it seems that segmentation is quite easy, while classification is hard, and it's more convenient to tune a classification model alone without worrying about detection, also pipeline is easier and more flexible.\n- A separate detection model, and then a classifier on single-character crops. This is probably the easiest approach to get a reasonable result, and makes it very easy to improve a classification model. Still I felt that using larger crops as inputs should provide better context for the model, so that it can see nearby symbols and would not suffer from not ideal crops. But it could be that classification on character crops can be better.\n\nNext come more details on each stage.\n\nSegmentation\n------------\n\nSegmentation into characters is done with a Faster-RCNN model with ``resnet152`` backbone trained with torchvision. Only one class is used, so it does not try to predict the character class. This model trains very fast and gives high quality boxes. Competition F1 metric (assuming\nperfect prediction for the classes) was around ~0.99 on validation.\n\nSome details:\n\n* torchvision detection pipeline was adapted,\n* ``resnet152`` backbone worked a bit better than default ``resnet50`` (even though it was not pre-trained on COCO, doing this would offer another small boost),\n* pipeline was modified to accept empty crops (crops without ground truth objects) to reduce amount of false positives,\n* it was trained on 512x384 crops, with page height around 1500 px, and full pages were used for inference,\n* augmentations used: scale, minor color augmentations (hue/saturation/value), Albumentations library was used.\n\nOverall many more improvements are possible here: using mmdetection, better models, pre-training on COCO, blending predictions from different folds for submission, TTA, separate model to discard out-of-page symbols, etc. Still it seemed that classification was more important.\n\nThis is implemented in https://github.com/lopuhin/kaggle-kuzushiji-2019/tree/master/kuzushiji/segment (which is based on reference torchvision detection code), dataset is defined in https://github.com/lopuhin/kaggle-kuzushiji-2019/blob/master/kuzushiji/segment/dataset.py\n\nHere are validation F1 scores (assuming perfect class prediction) for resnet50 pretrained on COCO (orange), resnet152 pretrained on ImageNet (green), and same but with a better training schedule (blue, used for the submission).\n\n\n\n\nClassification\n--------------\n\nClassification is performed by a model which gets as input a large crop from the image (512x768 from 2500-3000px high image) which contains multiple characters. It also receives as input boudning boxes predicted by segmentation model (these are out-of-fold predictions). This is similar to multi-class detection, but with frozen bounding boxes. ResNet base is used, ``layer4`` is discarded, features are extracted for each bounding box with ``roi_align`` from ``layer2`` and ``layer3`` and concatenated, and then passed into a classification head.\n\nSome details:\n\n* surprisingly, details such as architecture, backbone and learning regime made a lot of difference, much more than usual.\n* head with two fully-connected layers and two 0.5 dropout layers was used, and all details were important: features from roi pooling were very high-dimensional (more than 13k), first layer reduced this to 1024, and second layer performed final classification. Adding more layers or removing intermediate bottleneck reduced quality.\n* bigger backbones made a big difference, best model was the largest that could fit into 2080ti with a reasonable batch size: ``resnext101_32x8d_wsl`` from https://github.com/facebookresearch/WSL-Images\n* in order to train ``resnext101_32x8d_wsl`` on 2080ti, mixed precision training was required along with freezing first convolution and whole ``layer1`` (as I learned from Arthur Kuzin who did quite well in OpenImages, this is a trick used in mmdetection: https://github.com/open-mmlab/mmdetection/blob/6668bf0368b7ec6e88bc01aebdc281d2f79ef0cb/mmdet/models/backbones/resnet.py#L460)\n* another trick for reducing memory usage and making it train faster with cudnn.benchmark was limiting and bucketing number of targets in one batch.\n* model was very sensitive to hyperparameters such as crop size and shape and batch size (and gradient accumulation wasn't enough to fix this).\n* SGD with momentum performed significantly better than Adam, cosine schedule was used, weight decay was also quite important.\n* quite large scale and color augmentations were used: hue/saturation/value, random brighness, contrast and gamma, all from Albumentations library.\n* TTA (test-time-augmentation) of 4 different scales was used.\n* ``resnext101_32x8d_wsl`` took around 15 hours to train on one 2080ti.\n\nBest single model without pseudolabelling obtained public LB score of 0.935, although score varied quite a lot between folds, most folds were in 0.925 - 0.930 range. A blend of ``resnet152`` and ``resnext101_32x8d_wsl`` models across all folds scored 0.941 on the public LB.\n\nOverall, many improvement are possible here, from just using bigger models and freezing less layers, to more work on training schedule, augmentations, etc.\n\nSee https://github.com/lopuhin/kaggle-kuzushiji-2019/blob/master/kuzushiji/classify/main.py for the training script, https://github.com/lopuhin/kaggle-kuzushiji-2019/blob/master/kuzushiji/classify/models.py for the models, and https://github.com/lopuhin/kaggle-kuzushiji-2019/blob/master/kuzushiji/classify/dataset.py for the dataset and augmentations.\n\nHere are validation F1 scores of several classification models (including psedulabeling, see below). ``resnet50`` is orange, ``resnet152`` is blue, ``resnext101_32x8d_wsl`` is violet, the same fine-tuned on pseduo-labels is red, and the same trained from scratch on pseduo-labels is green.\n\n\n\n\nPseudolabelling\n---------------\n\nPseudolabelling is a technique where we take confident predictions of our model on test data, and add this to train dataset. Even though the model is already confident in such predictions, they are still useful and improve quality, because they allow the model to adapt better to different domain, as each book has it's own character and paper style, each author has different writing, etc.\n\nHere the simplest approach was chosen: most confident predictions were used for all test set, instead of splitting it by book. Top 80% most confident predictions from the blend were used, having accuracy >99% according to validation. Next, two kinds of models were trained (all based on ``resnext101_32x8d_wsl``):\n\n- models from previous step fine-tuned for 5 epochs (compared to 50 epochs for training from scratch) with starting learning rate 10x smaller than initial learning rate.\n- models trained from scratch with default settings.\n\nIn both cases, models used both train and test data for training. Best single fine-tuned model scored 0.939 on the public LB. Best single from-scratch model scored 0.944 on the public LB (0.945 on private LB).\n\nSee https://github.com/lopuhin/kaggle-kuzushiji-2019/blob/master/kuzushiji/classify/pseudolabel.py for a script which filters confident prediction. This data is added in the regular classification train script.\n\nSecond level model\n------------------\n\nA simple blend worked already quite well, giving 0.943 public LB (without pseudolabelled from-scratch models). Adjusting coefficients of the models didn't improve the validation score, even though ``resnext101_32x8d_wsl`` models were noticeably better.\n\nSince all models were trained across all folds, it was possible to train a second level model, a blend of LightGBM and XGBoost. This model was inspired by Pavel Ostyakov's solution to Cdiscount’s Image Classification Challenge, which was a classification problem with 5k classes:\nhttps://www.kaggle.com/c/cdiscount-image-classification-challenge/discussion/45733\n\nEach of 4 model kinds from classification contributed classes and scores of top-3 predictions as features. Also max overlap with other bboxes was added. Then for each of all classes in top-3 predictions, and for a ``seg_fp`` class, we created one row with an extra feature ``candidate``, which had a class as a value, and the target is binary: whether this candidate class was a true class which should be predicted. Then for each top-3 class, we added an extra binary feature which tells whether this class is a candidate class.\n\nHere is a simplified example with 1 model and top-2 predictions, all rows created for one character prediction (``seg_fp`` was encoded as -1, ``top0_s`` means ``top0_score``, ``top0_is_c`` means ``top0_is_candidate``)::\n\n top0_cls top1_cls top0_s top1_s candidate top0_is_c top1_is_c y\n 83 258 15.202 7.1246 83 True False True\n 83 258 15.202 7.1246 258 False True False\n 83 258 15.202 7.1246 -1 False False False\n\nXGBoost and LighGBM models were trained across all folds, and then blended. It was better to first apply models to fold predictions on test and then blend them.\n\nSuch blend gives 0.949 on public LB.\n\nI'm extremely bad at tuning such models, so there may be more improvements possible. Adjusting ``seg_fp`` ratio was tried and provided some boost on validation but didn't work on public LB.\n\nSecond level features are defined in https://github.com/lopuhin/kaggle-kuzushiji-2019/blob/master/kuzushiji/classify/level2_features.py and the models are built in https://github.com/lopuhin/kaggle-kuzushiji-2019/blob/master/kuzushiji/classify/level2.py\n\nLB score summary \n------------------\n\nScores on public LB (private LB scores are very well correlated):\n\n- ``resnet50``, fold0: 0.916\n- ``resnet152``, fold0: 0.926\n- ``resnet152``, fold4: 0.932 (same model as above, best fold)\n- ``resnext101_32x8d_wsl``, fold4: 0.934\n- ``resnext101_32x8d_wsl``, fold4, fine-tuned with pseudo-labels: 0.939\n- ``resnext101_32x8d_wsl``, fold4, re-trained with pseudo-labels: 0.944\n- second-level model on top of all folds and models: 0.949\n\nDiscarded ideas\n---------------\n\n* language model: a simple bi-LSTM language model was trained, but it achieved log loss of only ~4.5, while image-base model was at ~0.5, so it seemed that it would provide very little benefit. See https://github.com/lopuhin/kaggle-kuzushiji-2019/tree/master/kuzushiji/lm\n* kNN/metric learning: it's possible to use activations before the last layer as features, extract them from train and test, and then at inference time look closest (by cosine distance) example from train. This gave a minor boost over classification for single models, but inference time was quite high even with all optimizations, blending was less clear, so this was discarded. See https://github.com/lopuhin/kaggle-kuzushiji-2019/blob/master/kuzushiji/classify/knn.py\n* adding an LSTM on top of the image model, where LSTM would work over symbols in the image using the same sequencing approach from the language model - I tried this only briefly but it worked worse than regular classification.\n\nRunning\n----------\n\nSee https://github.com/lopuhin/kaggle-kuzushiji-2019#install and https://github.com/lopuhin/kaggle-kuzushiji-2019#run and also the script which in theory contains all steps https://github.com/lopuhin/kaggle-kuzushiji-2019/blob/master/run-all.sh but was not run in practice.\n\nHardware and libraries\n----------------------\n\nAlmost all models were trained on my home server with one 2080ti. ``resnet152`` classification models were trained on GCP with P100 GPUs as they required 16 GB of memory and I had some GCP credits. A few models towards the end were trained on vast.ai.\n\nAll models are written with pytorch, detection models are based on torchvision. Apex is used for mixed precision training, and Albumentations for augmentations.", "votes": 75 }, { "id": 649111, "postDate": "2019-10-15T01:16:19.340Z", "content": "Thank you so much for the solution!! And congratulations!
", "rawMarkdown": "Thank you so much for the solution!! And congratulations!", "votes": 5 }, { "id": 659378, "postDate": "2019-10-27T14:45:40.317Z", "content": "Updated:
\n\nCongratulations, and thank you for your sharing👍
", "rawMarkdown": "Congratulations, and thank you for your sharing👍 ", "votes": 1 }, { "id": 651108, "postDate": "2019-10-17T03:44:40.253Z", "content": "Thanks @lopuhin for sharing the great solution! Especially, the approach of classification is very unique! I'm curious to know the effect of large crop which contains multiple characters. Have you tried the other crop size (smaller than 512x768)?
", "rawMarkdown": "Thanks @lopuhin for sharing the great solution! Especially, the approach of classification is very unique! I'm curious to know the effect of large crop which contains multiple characters. Have you tried the other crop size (smaller than 512x768)?", "votes": 1, "replies": [ { "id": 651233, "postDate": "2019-10-17T07:58:15.110Z", "content": "Thank you! Yes I tried smaller crop sizes and they performed a bit worse (e.g. I tried 512x512 and maybe smaller at the start), but note that changing crop size also changes the number of targets in one batch, which might require learning rate or batch size adjustment - this is one of the drawbacks of this approach, that crop size is not an independent hyperparameter, and I didn't do much tuning here.
", "rawMarkdown": "Thank you! Yes I tried smaller crop sizes and they performed a bit worse (e.g. I tried 512x512 and maybe smaller at the start), but note that changing crop size also changes the number of targets in one batch, which might require learning rate or batch size adjustment - this is one of the drawbacks of this approach, that crop size is not an independent hyperparameter, and I didn't do much tuning here.", "votes": 2 }, { "id": 651275, "postDate": "2019-10-17T09:17:52.023Z", "content": "Thank you!
", "rawMarkdown": "Thank you!", "votes": 1 } ] }, { "id": 654020, "postDate": "2019-10-21T10:15:15.783Z", "content": "Congrats! 💪 🙌
", "rawMarkdown": "Congrats! 💪 🙌 ", "votes": 2 }, { "id": 652408, "postDate": "2019-10-18T20:15:52.013Z", "content": "Congrats on your scoring! Really enjoyed reading your solution, thank you for the work :)
", "rawMarkdown": "Congrats on your scoring! Really enjoyed reading your solution, thank you for the work :)", "votes": 2 }, { "id": 649445, "postDate": "2019-10-15T11:10:07.843Z", "content": "Great solution! But I am a bit confused with your classification approach.\n1. The input is an image that may contain multiple characters, is this the output of the segmentation or a randomized cropping that takes bounding box information for each individual character present in the cropped image?\n2. Would it be different to take the bounding box of a character and input directly to a classification network to then extract features from layer2 and layer3?
Sorry, I'm explaining this poorly - maybe this image would be helpful https://www.kaggle.com/c/kuzushiji-recognition/discussion/112204#647304 - creating a crop containing multiple images is not a problem, what is more important, how the network is trained - it uses the same approach as detection networks use, but with fixed bounding boxes. So the image is passed through a backbone (resnet), and this backbone outputs a tensor which has lower spacial resolution but has more channels (2048). Then for each bounding box, we extract features from this tensor with roi align (a modification of roi pooling), and for each bbox these features are passed into a classification head.
", "rawMarkdown": "Sorry, I'm explaining this poorly - maybe this image would be helpful https://www.kaggle.com/c/kuzushiji-recognition/discussion/112204#647304 - creating a crop containing multiple images is not a problem, what is more important, how the network is trained - it uses the same approach as detection networks use, but with fixed bounding boxes. So the image is passed through a backbone (resnet), and this backbone outputs a tensor which has lower spacial resolution but has more channels (2048). Then for each bounding box, we extract features from this tensor with roi align (a modification of roi pooling), and for each bbox these features are passed into a classification head.", "votes": 2 }, { "id": 649618, "postDate": "2019-10-15T15:22:52.097Z", "content": "Brilliant!
", "rawMarkdown": "Brilliant!", "votes": 1 } ] }, { "id": 649218, "postDate": "2019-10-15T04:55:33.787Z", "content": "Congratulations\nGreat Write-Up\nThanks for Sharing Your Approach & Insights, Code... @lopuhin
", "rawMarkdown": "Congratulations\nGreat Write-Up\nThanks for Sharing Your Approach & Insights, Code... @lopuhin ", "votes": 2 }, { "id": 649201, "postDate": "2019-10-15T04:09:35.603Z", "content": "Thank you so much for sharing this. I'm a newbie in this area and so happy to hear great solutions of you guys.
\n\nMind asking you some basic concepts of the augmentation when a class contains less than 10 images? How did you determine and handle the number of images to augment? or you see it as an outlier and leave them as they be?
", "rawMarkdown": "Thank you so much for sharing this. I'm a newbie in this area and so happy to hear great solutions of you guys. \n\nMind asking you some basic concepts of the augmentation when a class contains less than 10 images? How did you determine and handle the number of images to augment? or you see it as an outlier and leave them as they be? ", "votes": 2, "replies": [ { "id": 649333, "postDate": "2019-10-15T08:04:51.570Z", "content": "Thank you! Great point regarding images with small number of samples - I didn't do anything special with them, this could be another potential area for improvement.\nRegarding augmentations, they are applied dynamically during training, and the sampling worked in this way: first I sampled a random page, and then I sampled a random location inside this page to make a crop. Augmentations at test time were applied to the whole page.
", "rawMarkdown": "Thank you! Great point regarding images with small number of samples - I didn't do anything special with them, this could be another potential area for improvement.\nRegarding augmentations, they are applied dynamically during training, and the sampling worked in this way: first I sampled a random page, and then I sampled a random location inside this page to make a crop. Augmentations at test time were applied to the whole page.", "votes": 1 } ] }, { "id": 649368, "postDate": "2019-10-15T08:52:04.470Z", "content": "@lopuhin looks insane! I have to read it carefully and check the code.\nOur solution is very simple and we got over 0.9, but this is on a complete different level.
", "rawMarkdown": "@lopuhin looks insane! I have to read it carefully and check the code.\nOur solution is very simple and we got over 0.9, but this is on a complete different level.", "votes": 1, "replies": [ { "id": 649371, "postDate": "2019-10-15T08:59:24.910Z", "content": "Thanks! Looks forward to learning about your solution more, and for everyone else who would share more details.\nOne thing I was really unsure was (and still is), whether is would be better to just use single character crops for classification, maybe with some extra margin for context.
", "rawMarkdown": "Thanks! Looks forward to learning about your solution more, and for everyone else who would share more details.\nOne thing I was really unsure was (and still is), whether is would be better to just use single character crops for classification, maybe with some extra margin for context.", "votes": 2 } ] }, { "id": 675396, "postDate": "2019-11-18T03:05:42.730Z", "content": "Hello,
\n\nOrganizer here. If you wouldn't mind, can you say how much time your method takes for inference per-page-image (even a ballpark or rough estimate is fine)?
\n\nBest,
\n\nAlex.
", "rawMarkdown": "Hello, \n\nOrganizer here. If you wouldn't mind, can you say how much time your method takes for inference per-page-image (even a ballpark or rough estimate is fine)? \n\nBest, \n\nAlex. ", "replies": [ { "id": 675509, "postDate": "2019-11-18T06:30:24.787Z", "content": "Hello @thenuttynetter , \nHere are results for the best model, all times without TTA on 2080 ti, don't include image loading or resizing (but it would be quite small). All images are resized to max height 2528 px.
\n\nresnet152: ~230-40 ms per image (not using fp16).resnext101_32x8d_wsl: ~210-220 ms per image (using fp16 although without native extensions).Overall time would be around 440-460 ms per image, without pre/post processing, and without TTA.
", "rawMarkdown": "\nHello @thenuttynetter , \nHere are results for the best model, all times without TTA on 2080 ti, don't include image loading or resizing (but it would be quite small). All images are resized to max height 2528 px.\n\n1. Detection, using `resnet152`: ~230-40 ms per image (not using fp16).\n2. Classification, using `resnext101_32x8d_wsl`: ~210-220 ms per image (using fp16 although without native extensions).\n\nOverall time would be around 440-460 ms per image, without pre/post processing, and without TTA.", "votes": 1 } ] }, { "id": 657348, "postDate": "2019-10-25T03:52:12.477Z", "content": "Congrats!\nAmazing solution
", "rawMarkdown": "Congrats!\nAmazing solution" }, { "id": 655985, "postDate": "2019-10-23T18:52:33.200Z", "rawMarkdown": "", "isDeleted": true }, { "id": 650690, "postDate": "2019-10-16T15:48:30.420Z", "content": "Great solution, thank you!
", "rawMarkdown": "Great solution, thank you!", "votes": 1 } ], "comments": [ { "id": 649111, "author_name": "tkasasagi", "author_url": "", "post_date": "2019-10-15T01:16:19.340000", "content": "Thank you so much for the solution!! And congratulations!
", "votes": 5, "replies": [] }, { "id": 659378, "author_name": "Konstantin Lopukhin", "author_url": "", "post_date": "2019-10-27T14:45:40.317000", "content": "Updated:
\n\nCongratulations, and thank you for your sharing👍
", "votes": 1, "replies": [] }, { "id": 651108, "author_name": "K_mat", "author_url": "", "post_date": "2019-10-17T03:44:40.253000", "content": "Thanks @lopuhin for sharing the great solution! Especially, the approach of classification is very unique! I'm curious to know the effect of large crop which contains multiple characters. Have you tried the other crop size (smaller than 512x768)?
", "votes": 1, "replies": [ { "id": 651233, "author_name": "Konstantin Lopukhin", "author_url": "", "post_date": "2019-10-17T07:58:15.110000", "content": "Thank you! Yes I tried smaller crop sizes and they performed a bit worse (e.g. I tried 512x512 and maybe smaller at the start), but note that changing crop size also changes the number of targets in one batch, which might require learning rate or batch size adjustment - this is one of the drawbacks of this approach, that crop size is not an independent hyperparameter, and I didn't do much tuning here.
", "votes": 2, "replies": [] }, { "id": 651275, "author_name": "K_mat", "author_url": "", "post_date": "2019-10-17T09:17:52.023000", "content": "Thank you!
", "votes": 1, "replies": [] } ] }, { "id": 654020, "author_name": "Iván de Prado", "author_url": "", "post_date": "2019-10-21T10:15:15.783000", "content": "Congrats! 💪 🙌
", "votes": 2, "replies": [] }, { "id": 652408, "author_name": "Dmytro Danevskyi", "author_url": "", "post_date": "2019-10-18T20:15:52.013000", "content": "Congrats on your scoring! Really enjoyed reading your solution, thank you for the work :)
", "votes": 2, "replies": [] }, { "id": 649445, "author_name": "LintangSutawika", "author_url": "", "post_date": "2019-10-15T11:10:07.843000", "content": "Great solution! But I am a bit confused with your classification approach.\n1. The input is an image that may contain multiple characters, is this the output of the segmentation or a randomized cropping that takes bounding box information for each individual character present in the cropped image?\n2. Would it be different to take the bounding box of a character and input directly to a classification network to then extract features from layer2 and layer3?
Sorry, I'm explaining this poorly - maybe this image would be helpful https://www.kaggle.com/c/kuzushiji-recognition/discussion/112204#647304 - creating a crop containing multiple images is not a problem, what is more important, how the network is trained - it uses the same approach as detection networks use, but with fixed bounding boxes. So the image is passed through a backbone (resnet), and this backbone outputs a tensor which has lower spacial resolution but has more channels (2048). Then for each bounding box, we extract features from this tensor with roi align (a modification of roi pooling), and for each bbox these features are passed into a classification head.
", "votes": 2, "replies": [] }, { "id": 649618, "author_name": "LintangSutawika", "author_url": "", "post_date": "2019-10-15T15:22:52.097000", "content": "Brilliant!
", "votes": 1, "replies": [] } ] }, { "id": 649218, "author_name": "Ailurophile", "author_url": "", "post_date": "2019-10-15T04:55:33.787000", "content": "Congratulations\nGreat Write-Up\nThanks for Sharing Your Approach & Insights, Code... @lopuhin
", "votes": 2, "replies": [] }, { "id": 649201, "author_name": "Tanyapohn", "author_url": "", "post_date": "2019-10-15T04:09:35.603000", "content": "Thank you so much for sharing this. I'm a newbie in this area and so happy to hear great solutions of you guys.
\n\nMind asking you some basic concepts of the augmentation when a class contains less than 10 images? How did you determine and handle the number of images to augment? or you see it as an outlier and leave them as they be?
", "votes": 2, "replies": [ { "id": 649333, "author_name": "Konstantin Lopukhin", "author_url": "", "post_date": "2019-10-15T08:04:51.570000", "content": "Thank you! Great point regarding images with small number of samples - I didn't do anything special with them, this could be another potential area for improvement.\nRegarding augmentations, they are applied dynamically during training, and the sampling worked in this way: first I sampled a random page, and then I sampled a random location inside this page to make a crop. Augmentations at test time were applied to the whole page.
", "votes": 1, "replies": [] } ] }, { "id": 649368, "author_name": "Nanashi", "author_url": "", "post_date": "2019-10-15T08:52:04.470000", "content": "@lopuhin looks insane! I have to read it carefully and check the code.\nOur solution is very simple and we got over 0.9, but this is on a complete different level.
", "votes": 1, "replies": [ { "id": 649371, "author_name": "Konstantin Lopukhin", "author_url": "", "post_date": "2019-10-15T08:59:24.910000", "content": "Thanks! Looks forward to learning about your solution more, and for everyone else who would share more details.\nOne thing I was really unsure was (and still is), whether is would be better to just use single character crops for classification, maybe with some extra margin for context.
", "votes": 2, "replies": [] } ] }, { "id": 675396, "author_name": "TheNuttyNetter", "author_url": "", "post_date": "2019-11-18T03:05:42.730000", "content": "Hello,
\n\nOrganizer here. If you wouldn't mind, can you say how much time your method takes for inference per-page-image (even a ballpark or rough estimate is fine)?
\n\nBest,
\n\nAlex.
", "votes": 0, "replies": [ { "id": 675509, "author_name": "Konstantin Lopukhin", "author_url": "", "post_date": "2019-11-18T06:30:24.787000", "content": "Hello @thenuttynetter , \nHere are results for the best model, all times without TTA on 2080 ti, don't include image loading or resizing (but it would be quite small). All images are resized to max height 2528 px.
\n\nresnet152: ~230-40 ms per image (not using fp16).resnext101_32x8d_wsl: ~210-220 ms per image (using fp16 although without native extensions).Overall time would be around 440-460 ms per image, without pre/post processing, and without TTA.
", "votes": 1, "replies": [] } ] }, { "id": 657348, "author_name": "wakas", "author_url": "", "post_date": "2019-10-25T03:52:12.477000", "content": "Congrats!\nAmazing solution
", "votes": 0, "replies": [] }, { "id": 655985, "author_name": "", "author_url": "", "post_date": "2019-10-23T18:52:33.200000", "content": "", "votes": 0, "replies": [] }, { "id": 650690, "author_name": "Ma Yuxuan", "author_url": "", "post_date": "2019-10-16T15:48:30.420000", "content": "Great solution, thank you!
", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "649071": "Thanks to all organizers and Kaggle for such an interesting dataset and competition, and congrats to everyone who finished 🎉 \n\nBelow is an overview of my solution. Code is at https://github.com/lopuhin/kaggle-kuzushiji-2019, model weights for the best model are at https://github.com/lopuhin/kaggle-kuzushiji-2019/releases/tag/v1.0\n\nGeneral approach is as follows:\n\n- Dataset is split into 5 folds by book.\n- Class-agnostic bounding boxes are predicted for all characters using an object detection network (with ``resnet152`` backbone pretrained on ImageNet). Out-of-fold predictions are obtained for all 5 folds.\n- A \"classification\" model is trained using OOF detection predictions. An extra class ``seg_fp`` (segmentation false-positive) is added for bounding boxes which have low overlap with ground truth boxes, so classification model can correct errors of segmentation model. Classification model is trained on all folds. Models with ``resnet152`` and ``resnext101_32x8d_wsl`` backbones are used, they are trained on large crops containing multiple symbols, using FPN and roi align with a classification head.\n- Pseudolabelling is performed, in OCR terms this is similar to \"writer adaptation\", although here it is applied to the whole test for simplicity.\n- A second level model is trained on classification predictions, which creates the final submission.\n\nWhy such approach was chosen? There are two other candidate approaches:\n\n- End-to-end model which does detection and classification (e.g. Faster-RCNN). This may be possible with some effort, but here it seems that segmentation is quite easy, while classification is hard, and it's more convenient to tune a classification model alone without worrying about detection, also pipeline is easier and more flexible.\n- A separate detection model, and then a classifier on single-character crops. This is probably the easiest approach to get a reasonable result, and makes it very easy to improve a classification model. Still I felt that using larger crops as inputs should provide better context for the model, so that it can see nearby symbols and would not suffer from not ideal crops. But it could be that classification on character crops can be better.\n\nNext come more details on each stage.\n\nSegmentation\n------------\n\nSegmentation into characters is done with a Faster-RCNN model with ``resnet152`` backbone trained with torchvision. Only one class is used, so it does not try to predict the character class. This model trains very fast and gives high quality boxes. Competition F1 metric (assuming\nperfect prediction for the classes) was around ~0.99 on validation.\n\nSome details:\n\n* torchvision detection pipeline was adapted,\n* ``resnet152`` backbone worked a bit better than default ``resnet50`` (even though it was not pre-trained on COCO, doing this would offer another small boost),\n* pipeline was modified to accept empty crops (crops without ground truth objects) to reduce amount of false positives,\n* it was trained on 512x384 crops, with page height around 1500 px, and full pages were used for inference,\n* augmentations used: scale, minor color augmentations (hue/saturation/value), Albumentations library was used.\n\nOverall many more improvements are possible here: using mmdetection, better models, pre-training on COCO, blending predictions from different folds for submission, TTA, separate model to discard out-of-page symbols, etc. Still it seemed that classification was more important.\n\nThis is implemented in https://github.com/lopuhin/kaggle-kuzushiji-2019/tree/master/kuzushiji/segment (which is based on reference torchvision detection code), dataset is defined in https://github.com/lopuhin/kaggle-kuzushiji-2019/blob/master/kuzushiji/segment/dataset.py\n\nHere are validation F1 scores (assuming perfect class prediction) for resnet50 pretrained on COCO (orange), resnet152 pretrained on ImageNet (green), and same but with a better training schedule (blue, used for the submission).\n\n\n\n\nClassification\n--------------\n\nClassification is performed by a model which gets as input a large crop from the image (512x768 from 2500-3000px high image) which contains multiple characters. It also receives as input boudning boxes predicted by segmentation model (these are out-of-fold predictions). This is similar to multi-class detection, but with frozen bounding boxes. ResNet base is used, ``layer4`` is discarded, features are extracted for each bounding box with ``roi_align`` from ``layer2`` and ``layer3`` and concatenated, and then passed into a classification head.\n\nSome details:\n\n* surprisingly, details such as architecture, backbone and learning regime made a lot of difference, much more than usual.\n* head with two fully-connected layers and two 0.5 dropout layers was used, and all details were important: features from roi pooling were very high-dimensional (more than 13k), first layer reduced this to 1024, and second layer performed final classification. Adding more layers or removing intermediate bottleneck reduced quality.\n* bigger backbones made a big difference, best model was the largest that could fit into 2080ti with a reasonable batch size: ``resnext101_32x8d_wsl`` from https://github.com/facebookresearch/WSL-Images\n* in order to train ``resnext101_32x8d_wsl`` on 2080ti, mixed precision training was required along with freezing first convolution and whole ``layer1`` (as I learned from Arthur Kuzin who did quite well in OpenImages, this is a trick used in mmdetection: https://github.com/open-mmlab/mmdetection/blob/6668bf0368b7ec6e88bc01aebdc281d2f79ef0cb/mmdet/models/backbones/resnet.py#L460)\n* another trick for reducing memory usage and making it train faster with cudnn.benchmark was limiting and bucketing number of targets in one batch.\n* model was very sensitive to hyperparameters such as crop size and shape and batch size (and gradient accumulation wasn't enough to fix this).\n* SGD with momentum performed significantly better than Adam, cosine schedule was used, weight decay was also quite important.\n* quite large scale and color augmentations were used: hue/saturation/value, random brighness, contrast and gamma, all from Albumentations library.\n* TTA (test-time-augmentation) of 4 different scales was used.\n* ``resnext101_32x8d_wsl`` took around 15 hours to train on one 2080ti.\n\nBest single model without pseudolabelling obtained public LB score of 0.935, although score varied quite a lot between folds, most folds were in 0.925 - 0.930 range. A blend of ``resnet152`` and ``resnext101_32x8d_wsl`` models across all folds scored 0.941 on the public LB.\n\nOverall, many improvement are possible here, from just using bigger models and freezing less layers, to more work on training schedule, augmentations, etc.\n\nSee https://github.com/lopuhin/kaggle-kuzushiji-2019/blob/master/kuzushiji/classify/main.py for the training script, https://github.com/lopuhin/kaggle-kuzushiji-2019/blob/master/kuzushiji/classify/models.py for the models, and https://github.com/lopuhin/kaggle-kuzushiji-2019/blob/master/kuzushiji/classify/dataset.py for the dataset and augmentations.\n\nHere are validation F1 scores of several classification models (including psedulabeling, see below). ``resnet50`` is orange, ``resnet152`` is blue, ``resnext101_32x8d_wsl`` is violet, the same fine-tuned on pseduo-labels is red, and the same trained from scratch on pseduo-labels is green.\n\n\n\n\nPseudolabelling\n---------------\n\nPseudolabelling is a technique where we take confident predictions of our model on test data, and add this to train dataset. Even though the model is already confident in such predictions, they are still useful and improve quality, because they allow the model to adapt better to different domain, as each book has it's own character and paper style, each author has different writing, etc.\n\nHere the simplest approach was chosen: most confident predictions were used for all test set, instead of splitting it by book. Top 80% most confident predictions from the blend were used, having accuracy >99% according to validation. Next, two kinds of models were trained (all based on ``resnext101_32x8d_wsl``):\n\n- models from previous step fine-tuned for 5 epochs (compared to 50 epochs for training from scratch) with starting learning rate 10x smaller than initial learning rate.\n- models trained from scratch with default settings.\n\nIn both cases, models used both train and test data for training. Best single fine-tuned model scored 0.939 on the public LB. Best single from-scratch model scored 0.944 on the public LB (0.945 on private LB).\n\nSee https://github.com/lopuhin/kaggle-kuzushiji-2019/blob/master/kuzushiji/classify/pseudolabel.py for a script which filters confident prediction. This data is added in the regular classification train script.\n\nSecond level model\n------------------\n\nA simple blend worked already quite well, giving 0.943 public LB (without pseudolabelled from-scratch models). Adjusting coefficients of the models didn't improve the validation score, even though ``resnext101_32x8d_wsl`` models were noticeably better.\n\nSince all models were trained across all folds, it was possible to train a second level model, a blend of LightGBM and XGBoost. This model was inspired by Pavel Ostyakov's solution to Cdiscount’s Image Classification Challenge, which was a classification problem with 5k classes:\nhttps://www.kaggle.com/c/cdiscount-image-classification-challenge/discussion/45733\n\nEach of 4 model kinds from classification contributed classes and scores of top-3 predictions as features. Also max overlap with other bboxes was added. Then for each of all classes in top-3 predictions, and for a ``seg_fp`` class, we created one row with an extra feature ``candidate``, which had a class as a value, and the target is binary: whether this candidate class was a true class which should be predicted. Then for each top-3 class, we added an extra binary feature which tells whether this class is a candidate class.\n\nHere is a simplified example with 1 model and top-2 predictions, all rows created for one character prediction (``seg_fp`` was encoded as -1, ``top0_s`` means ``top0_score``, ``top0_is_c`` means ``top0_is_candidate``)::\n\n top0_cls top1_cls top0_s top1_s candidate top0_is_c top1_is_c y\n 83 258 15.202 7.1246 83 True False True\n 83 258 15.202 7.1246 258 False True False\n 83 258 15.202 7.1246 -1 False False False\n\nXGBoost and LighGBM models were trained across all folds, and then blended. It was better to first apply models to fold predictions on test and then blend them.\n\nSuch blend gives 0.949 on public LB.\n\nI'm extremely bad at tuning such models, so there may be more improvements possible. Adjusting ``seg_fp`` ratio was tried and provided some boost on validation but didn't work on public LB.\n\nSecond level features are defined in https://github.com/lopuhin/kaggle-kuzushiji-2019/blob/master/kuzushiji/classify/level2_features.py and the models are built in https://github.com/lopuhin/kaggle-kuzushiji-2019/blob/master/kuzushiji/classify/level2.py\n\nLB score summary \n------------------\n\nScores on public LB (private LB scores are very well correlated):\n\n- ``resnet50``, fold0: 0.916\n- ``resnet152``, fold0: 0.926\n- ``resnet152``, fold4: 0.932 (same model as above, best fold)\n- ``resnext101_32x8d_wsl``, fold4: 0.934\n- ``resnext101_32x8d_wsl``, fold4, fine-tuned with pseudo-labels: 0.939\n- ``resnext101_32x8d_wsl``, fold4, re-trained with pseudo-labels: 0.944\n- second-level model on top of all folds and models: 0.949\n\nDiscarded ideas\n---------------\n\n* language model: a simple bi-LSTM language model was trained, but it achieved log loss of only ~4.5, while image-base model was at ~0.5, so it seemed that it would provide very little benefit. See https://github.com/lopuhin/kaggle-kuzushiji-2019/tree/master/kuzushiji/lm\n* kNN/metric learning: it's possible to use activations before the last layer as features, extract them from train and test, and then at inference time look closest (by cosine distance) example from train. This gave a minor boost over classification for single models, but inference time was quite high even with all optimizations, blending was less clear, so this was discarded. See https://github.com/lopuhin/kaggle-kuzushiji-2019/blob/master/kuzushiji/classify/knn.py\n* adding an LSTM on top of the image model, where LSTM would work over symbols in the image using the same sequencing approach from the language model - I tried this only briefly but it worked worse than regular classification.\n\nRunning\n----------\n\nSee https://github.com/lopuhin/kaggle-kuzushiji-2019#install and https://github.com/lopuhin/kaggle-kuzushiji-2019#run and also the script which in theory contains all steps https://github.com/lopuhin/kaggle-kuzushiji-2019/blob/master/run-all.sh but was not run in practice.\n\nHardware and libraries\n----------------------\n\nAlmost all models were trained on my home server with one 2080ti. ``resnet152`` classification models were trained on GCP with P100 GPUs as they required 16 GB of memory and I had some GCP credits. A few models towards the end were trained on vast.ai.\n\nAll models are written with pytorch, detection models are based on torchvision. Apex is used for mixed precision training, and Albumentations for augmentations.", "649111": "Thank you so much for the solution!! And congratulations!", "659378": "Updated:\n\n- added links to the source code for each section\n- added some charts with validation F1 for different models\n- added more single-model scores and a summary with LB scores at the end\n- model weights", "652660": "Congratulations, and thank you for your sharing👍 ", "651108": "Thanks @lopuhin for sharing the great solution! Especially, the approach of classification is very unique! I'm curious to know the effect of large crop which contains multiple characters. Have you tried the other crop size (smaller than 512x768)?", "654020": "Congrats! 💪 🙌 ", "652408": "Congrats on your scoring! Really enjoyed reading your solution, thank you for the work :)", "649445": "Great solution! But I am a bit confused with your classification approach.\n1. The input is an image that may contain multiple characters, is this the output of the segmentation or a randomized cropping that takes bounding box information for each individual character present in the cropped image?\n2. Would it be different to take the bounding box of a character and input directly to a classification network to then extract features from `layer2` and `layer3`? \n", "649218": "Congratulations\nGreat Write-Up\nThanks for Sharing Your Approach & Insights, Code... @lopuhin ", "649201": "Thank you so much for sharing this. I'm a newbie in this area and so happy to hear great solutions of you guys. \n\nMind asking you some basic concepts of the augmentation when a class contains less than 10 images? How did you determine and handle the number of images to augment? or you see it as an outlier and leave them as they be? ", "649368": "@lopuhin looks insane! I have to read it carefully and check the code.\nOur solution is very simple and we got over 0.9, but this is on a complete different level.", "675396": "Hello, \n\nOrganizer here. If you wouldn't mind, can you say how much time your method takes for inference per-page-image (even a ballpark or rough estimate is fine)? \n\nBest, \n\nAlex. ", "657348": "Congrats!\nAmazing solution", "655985": "", "650690": "Great solution, thank you!" } }