Thanks to the Center for Open Data in the Humanities and Kaggle for this great challenge. Congratulations to the winners and everyone who had fun in this challenge. Here are a few details of my approach:
\n\nI already created a port of CenterNet for Keras: https://github.com/see--/keras-centernet but I never really trained a model with this repo. So I decided to get the training running as well. I reimplemented CenterNet from scratch with the following modifications:
\n\nReplace the Hourglass network with ResNet50 and ResNet101 + FPN decoder. Hourglass is too demanding in terms of hardware. You could probably still get better results.
Use a simpler loss function. I replaced the penalty-reduced pixel-wise logistic regression with focal loss by binary cross-entropy. This worked well for me and removed a few hyperparameters.
I also removed the heads that regressed the height & width and the x & y offsets. You can regress them easily with L2 loss but the metric only wants centers. We also don't need to be more accurate than +-2 pixels (which is what the offsets are used for). This gave a little speed-up.
CenterNet uses a gaussian around the the centers. I am not sure this is needed. I still use the following 3x3 kernel: \n\npos_kernel = np.float32([[0.5, 0.75, 0.5], [0.75, 1.0, 0.75], [0.5, 0.75, 0.5]])\n\nBut using only the exact center gave very similar results (i.e. \npos_kernel = np.float32([[0.0, 0.0, 0.0],\n [0.0, 1.0, 0.0],\n [0.0, 0.0, 0.0]])\n). One just needs to reduce the threshold.
A single ResNet101 that I trained for 125 epochs could reach: 0.938. I used the improved version from the Bag of Tricks for Image Classification paper: https://arxiv.org/abs/1812.01187. Thanks to timm this is really simple: timm.create_model('gluon_resnet101_v1d', pretrained=pretrained).
Albumentations was also super useful. Getting the geometric transforms for bounding boxes correct is not trivial. I used the following augmentations (without validation):\n\n self.aug = Compose([\n ShiftScaleRotate(p=0.9, rotate_limit=10,\n scale_limit=0.2, border_mode=cv2.BORDER_CONSTANT),\n RandomCrop(512, 512, p=1.0),\n ToGray(),\n CLAHE(),\n GaussNoise(),\n GaussianBlur(),\n RandomBrightnessContrast(),\n RandomGamma(),\n RGBShift(),\n HueSaturationValue(),\n ], bbox_params=BboxParams(format='coco', min_visibility=0.75))\n
A high min_visibility is set to not predict partial letters (that is how the data was labeled).
Finally, scale TTA (with scales [0.75, 1.0, 1.25]) helped a bit 0.938 -> 0.939.
What did not work
\n\nMulti-stage approaches with separate detection and classification. I only got ~0.85. One stage worked always better. I think this makes sense as the model can learn some kind of language model from the images.
Increasing the resolution even further. I am running predictions at 1536x1536 resolution but I didn't see any improvement from higher resolutions.
Creating more letters. This was a popular trick in the whale competition. We double the number of classes by adding flipped versions. However, the validation score was about the same so I removed it as it added code & complexity.
Hardware
\n\nThe final model can be trained in ~8 hours on a single V100 VM. I mostly used vast.ai and GCP.
\n\nThanks for reading. You can find the code and trained weights on GitHub:
\n\nThanks to the Center for Open Data in the Humanities and Kaggle for this great challenge. Congratulations to the winners and everyone who had fun in this challenge. Here are a few details of my approach:
\n\nI already created a port of CenterNet for Keras: https://github.com/see--/keras-centernet but I never really trained a model with this repo. So I decided to get the training running as well. I reimplemented CenterNet from scratch with the following modifications:
\n\nReplace the Hourglass network with ResNet50 and ResNet101 + FPN decoder. Hourglass is too demanding in terms of hardware. You could probably still get better results.
Use a simpler loss function. I replaced the penalty-reduced pixel-wise logistic regression with focal loss by binary cross-entropy. This worked well for me and removed a few hyperparameters.
I also removed the heads that regressed the height & width and the x & y offsets. You can regress them easily with L2 loss but the metric only wants centers. We also don't need to be more accurate than +-2 pixels (which is what the offsets are used for). This gave a little speed-up.
CenterNet uses a gaussian around the the centers. I am not sure this is needed. I still use the following 3x3 kernel: \n\npos_kernel = np.float32([[0.5, 0.75, 0.5], [0.75, 1.0, 0.75], [0.5, 0.75, 0.5]])\n\nBut using only the exact center gave very similar results (i.e. \npos_kernel = np.float32([[0.0, 0.0, 0.0],\n [0.0, 1.0, 0.0],\n [0.0, 0.0, 0.0]])\n). One just needs to reduce the threshold.
A single ResNet101 that I trained for 125 epochs could reach: 0.938. I used the improved version from the Bag of Tricks for Image Classification paper: https://arxiv.org/abs/1812.01187. Thanks to timm this is really simple: timm.create_model('gluon_resnet101_v1d', pretrained=pretrained).
Albumentations was also super useful. Getting the geometric transforms for bounding boxes correct is not trivial. I used the following augmentations (without validation):\n\n self.aug = Compose([\n ShiftScaleRotate(p=0.9, rotate_limit=10,\n scale_limit=0.2, border_mode=cv2.BORDER_CONSTANT),\n RandomCrop(512, 512, p=1.0),\n ToGray(),\n CLAHE(),\n GaussNoise(),\n GaussianBlur(),\n RandomBrightnessContrast(),\n RandomGamma(),\n RGBShift(),\n HueSaturationValue(),\n ], bbox_params=BboxParams(format='coco', min_visibility=0.75))\n
A high min_visibility is set to not predict partial letters (that is how the data was labeled).
Finally, scale TTA (with scales [0.75, 1.0, 1.25]) helped a bit 0.938 -> 0.939.
What did not work
\n\nMulti-stage approaches with separate detection and classification. I only got ~0.85. One stage worked always better. I think this makes sense as the model can learn some kind of language model from the images.
Increasing the resolution even further. I am running predictions at 1536x1536 resolution but I didn't see any improvement from higher resolutions.
Creating more letters. This was a popular trick in the whale competition. We double the number of classes by adding flipped versions. However, the validation score was about the same so I removed it as it added code & complexity.
Hardware
\n\nThe final model can be trained in ~8 hours on a single V100 VM. I mostly used vast.ai and GCP.
\n\nThanks for reading. You can find the code and trained weights on GitHub:
\n\nThanks for sharing, this is a really elegant solution and a very strong single model score, congrats! I'm curious what validation split did you use?
", "rawMarkdown": "Thanks for sharing, this is a really elegant solution and a very strong single model score, congrats! I'm curious what validation split did you use?", "votes": null }, { "id": "649438", "postDate": "10/15/2019 10:49:39", "content": "Thanks! I used 20% (randomly selected) as validation data. The final model is trained with all data. This gave a big improvement: From 0.907 -> 0.938. Probably because some letters never occurred during training with validation data.
Thank you so much!!!
", "rawMarkdown": "Thank you so much!!!", "votes": null }, { "id": "649596", "postDate": "10/15/2019 14:58:26", "content": "Congratulations\nGreat Write-Up\nThanks for Sharing your Approach & Insights.. @seesee
", "rawMarkdown": "Congratulations\nGreat Write-Up\nThanks for Sharing your Approach & Insights.. @seesee", "votes": null }, { "id": "675400", "postDate": "11/18/2019 03:09:11", "content": "Hello,
\n\nOrganizer here. If you wouldn't mind, can you say how much time your method takes for inference per-page-image (even a ballpark or rough estimate is fine)?
\n\nBest,
\n\nAlex.
", "rawMarkdown": "Hello, \n\nOrganizer here. If you wouldn't mind, can you say how much time your method takes for inference per-page-image (even a ballpark or rough estimate is fine)? \n\nBest, \n\nAlex.", "votes": null }, { "id": "675692", "postDate": "11/18/2019 12:35:06", "content": "@seesee Thanks for sharing ....
", "rawMarkdown": "seesee Thanks for sharing ....", "votes": null }, { "id": "688281", "postDate": "12/05/2019 12:25:52", "content": "'Hourglass is too demanding in terms of hardware', if I just use free kernel and pre-trained weights to train just part of network, can it work? @seesee
", "rawMarkdown": "'Hourglass is too demanding in terms of hardware', if I just use free kernel and pre-trained weights to train just part of network, can it work? @seesee", "votes": null }, { "id": "706233", "postDate": "12/30/2019 05:20:09", "content": "Hi see--:\nI'm using your keras centernet, it's wonderful, may I ask if this is the weight you train or get from original repo? @seesee
\n\nCTDET_COCO_WEIGHTS_PATH =('https://github.com/see-/kerascenternet/'releases/download/0.1.0/ctdet_coco_hg.hdf5')
", "rawMarkdown": "Hi see--:\nI'm using your keras centernet, it's wonderful, may I ask if this is the weight you train or get from original repo? @seesee \n\nCTDET_COCO_WEIGHTS_PATH =('https://github.com/see-/kerascenternet/'releases/download/0.1.0/ctdet_coco_hg.hdf5')", "votes": null }, { "id": "706646", "postDate": "12/30/2019 16:49:10", "content": "Hi @diegojohnson,
\n\nI glad that you find it useful ;)
\n\n\n\n\n'Hourglass is too demanding in terms of hardware', if I just use free kernel and pre-trained weights to train just part of network, can it work
\n
From my experience, training the whole model is always better than freezing parts.
\n\n\n\n\nI'm using your keras centernet, it's wonderful, may I ask if this is the weight you train or get from original repo?
\n
The weights are from the original repo. I hope I find some time to update the code to tf 2.0 and add the training part. It's much more efficient and simpler to use with keras.
", "rawMarkdown": "Hi @diegojohnson,\n\nI glad that you find it useful ;)\n\n> 'Hourglass is too demanding in terms of hardware', if I just use free kernel and pre-trained weights to train just part of network, can it work\n\nFrom my experience, training the whole model is always better than freezing parts.\n\n> I'm using your keras centernet, it's wonderful, may I ask if this is the weight you train or get from original repo?\n\nThe weights are from the original repo. I hope I find some time to update the code to tf 2.0 and add the training part. It's much more efficient and simpler to use with keras.", "votes": null }, { "id": "706648", "postDate": "12/30/2019 16:52:31", "content": "A rough guess is faster than 10 FPS on a 1080 TI without TTA.
", "rawMarkdown": "A rough guess is faster than 10 FPS on a 1080 TI without TTA.", "votes": null }, { "id": "706657", "postDate": "12/30/2019 17:08:22", "content": "thanks very much.😃
", "rawMarkdown": "thanks very much.😃", "votes": null } ], "comments": [ { "id": 649431, "author_name": "lopuhin", "author_url": "", "post_date": "10/15/2019 10:41:22", "content": "Thanks for sharing, this is a really elegant solution and a very strong single model score, congrats! I'm curious what validation split did you use?
", "votes": null, "replies": [ { "id": 649438, "author_name": "seesee", "author_url": "", "post_date": "10/15/2019 10:49:39", "content": "Thanks! I used 20% (randomly selected) as validation data. The final model is trained with all data. This gave a big improvement: From 0.907 -> 0.938. Probably because some letters never occurred during training with validation data.
Thank you so much!!!
", "votes": null, "replies": [] }, { "id": 649596, "author_name": "veeralakrishna", "author_url": "", "post_date": "10/15/2019 14:58:26", "content": "Congratulations\nGreat Write-Up\nThanks for Sharing your Approach & Insights.. @seesee
", "votes": null, "replies": [] }, { "id": 675400, "author_name": "thenuttynetter", "author_url": "", "post_date": "11/18/2019 03:09:11", "content": "Hello,
\n\nOrganizer here. If you wouldn't mind, can you say how much time your method takes for inference per-page-image (even a ballpark or rough estimate is fine)?
\n\nBest,
\n\nAlex.
", "votes": null, "replies": [ { "id": 706648, "author_name": "seesee", "author_url": "", "post_date": "12/30/2019 16:52:31", "content": "A rough guess is faster than 10 FPS on a 1080 TI without TTA.
", "votes": null, "replies": [] } ] }, { "id": 675692, "author_name": "saurav9786", "author_url": "", "post_date": "11/18/2019 12:35:06", "content": "@seesee Thanks for sharing ....
", "votes": null, "replies": [] }, { "id": 688281, "author_name": "diegojohnson", "author_url": "", "post_date": "12/05/2019 12:25:52", "content": "'Hourglass is too demanding in terms of hardware', if I just use free kernel and pre-trained weights to train just part of network, can it work? @seesee
", "votes": null, "replies": [] }, { "id": 706233, "author_name": "diegojohnson", "author_url": "", "post_date": "12/30/2019 05:20:09", "content": "Hi see--:\nI'm using your keras centernet, it's wonderful, may I ask if this is the weight you train or get from original repo? @seesee
\n\nCTDET_COCO_WEIGHTS_PATH =('https://github.com/see-/kerascenternet/'releases/download/0.1.0/ctdet_coco_hg.hdf5')
", "votes": null, "replies": [ { "id": 706646, "author_name": "seesee", "author_url": "", "post_date": "12/30/2019 16:49:10", "content": "Hi @diegojohnson,
\n\nI glad that you find it useful ;)
\n\n\n\n\n'Hourglass is too demanding in terms of hardware', if I just use free kernel and pre-trained weights to train just part of network, can it work
\n
From my experience, training the whole model is always better than freezing parts.
\n\n\n\n\nI'm using your keras centernet, it's wonderful, may I ask if this is the weight you train or get from original repo?
\n
The weights are from the original repo. I hope I find some time to update the code to tf 2.0 and add the training part. It's much more efficient and simpler to use with keras.
", "votes": null, "replies": [] }, { "id": 706657, "author_name": "diegojohnson", "author_url": "", "post_date": "12/30/2019 17:08:22", "content": "thanks very much.😃
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "649386": "Thanks to the Center for Open Data in the Humanities and Kaggle for this great challenge. Congratulations to the winners and everyone who had fun in this challenge. Here are a few details of my approach:\n\nI already created a port of [CenterNet](https://arxiv.org/abs/1904.07850) for Keras: https://github.com/see--/keras-centernet but I never really trained a model with this repo. So I decided to get the training running as well. I reimplemented CenterNet from scratch with the following modifications:\n\n* Replace the Hourglass network with ResNet50 and ResNet101 + [FPN](http://presentations.cocodataset.org/COCO17-Stuff-FAIR.pdf) decoder. Hourglass is too demanding in terms of hardware. You could probably still get better results.\n\n* Use a simpler loss function. I replaced the penalty-reduced pixel-wise logistic regression with focal loss by binary cross-entropy. This worked well for me and removed a few hyperparameters.\n\n* I also removed the heads that regressed the height & width and the x & y offsets. You can regress them easily with L2 loss but the metric only wants centers. We also don't need to be more accurate than +-2 pixels (which is what the offsets are used for). This gave a little speed-up.\n\n* CenterNet uses a gaussian around the the centers. I am not sure this is needed. I still use the following 3x3 kernel: \n```\npos_kernel = np.float32([[0.5, 0.75, 0.5], [0.75, 1.0, 0.75], [0.5, 0.75, 0.5]])\n```\nBut using only the exact center gave very similar results (i.e. ```\npos_kernel = np.float32([[0.0, 0.0, 0.0],\n [0.0, 1.0, 0.0],\n [0.0, 0.0, 0.0]])\n```). One just needs to reduce the threshold.\n\nA single ResNet101 that I trained for 125 epochs could reach: `0.938`. I used the improved version from the Bag of Tricks for Image Classification paper: https://arxiv.org/abs/1812.01187. Thanks to `timm` this is really simple: `timm.create_model('gluon_resnet101_v1d', pretrained=pretrained)`.\n\nAlbumentations was also super useful. Getting the geometric transforms for bounding boxes correct is [not trivial](https://github.com/poodarchu/learn_aug_for_object_detection.numpy). I used the following augmentations (without validation):\n```\n self.aug = Compose([\n ShiftScaleRotate(p=0.9, rotate_limit=10,\n scale_limit=0.2, border_mode=cv2.BORDER_CONSTANT),\n RandomCrop(512, 512, p=1.0),\n ToGray(),\n CLAHE(),\n GaussNoise(),\n GaussianBlur(),\n RandomBrightnessContrast(),\n RandomGamma(),\n RGBShift(),\n HueSaturationValue(),\n ], bbox_params=BboxParams(format='coco', min_visibility=0.75))\n```\n\nA high `min_visibility` is set to not predict partial letters (that is how the data was labeled).\n\nFinally, scale TTA (with scales `[0.75, 1.0, 1.25]`) helped a bit `0.938` -> `0.939`.\n\n**What did not work**\n\n* Multi-stage approaches with separate detection and classification. I only got ~`0.85`. One stage worked always better. I think this makes sense as the model can learn some kind of language model from the images.\n\n* Increasing the resolution even further. I am running predictions at 1536x1536 resolution but I didn't see any improvement from higher resolutions.\n\n* Creating more letters. This was a popular trick in the whale competition. We double the number of classes by adding flipped versions. However, the validation score was about the same so I removed it as it added code & complexity.\n\n**Hardware**\n\nThe final model can be trained in ~8 hours on a single V100 VM. I mostly used vast.ai and GCP.\n\nThanks for reading. You can find the code and trained weights on GitHub:\n\n* https://github.com/see--/kuzushiji-recognition\n* https://github.com/see--/kuzushiji-recognition/releases/download/v0.1/f00-ep-0125-val_hm_acc-0.9944-val_classes_acc-0.4986.pth", "649431": "Thanks for sharing, this is a really elegant solution and a very strong single model score, congrats! I'm curious what validation split did you use?", "649438": "Thanks! I used 20% (randomly selected) as validation data. The final model is trained with all data. This gave a big improvement: From `0.907` -> `0.938`. Probably because some letters never occurred during training with validation data.", "649450": "Thank you so much!!!", "649596": "Congratulations\nGreat Write-Up\nThanks for Sharing your Approach & Insights.. @seesee", "675400": "Hello, \n\nOrganizer here. If you wouldn't mind, can you say how much time your method takes for inference per-page-image (even a ballpark or rough estimate is fine)? \n\nBest, \n\nAlex.", "675692": "seesee Thanks for sharing ....", "688281": "'Hourglass is too demanding in terms of hardware', if I just use free kernel and pre-trained weights to train just part of network, can it work? @seesee", "706233": "Hi see--:\nI'm using your keras centernet, it's wonderful, may I ask if this is the weight you train or get from original repo? @seesee \n\nCTDET_COCO_WEIGHTS_PATH =('https://github.com/see-/kerascenternet/'releases/download/0.1.0/ctdet_coco_hg.hdf5')", "706646": "Hi @diegojohnson,\n\nI glad that you find it useful ;)\n\n> 'Hourglass is too demanding in terms of hardware', if I just use free kernel and pre-trained weights to train just part of network, can it work\n\nFrom my experience, training the whole model is always better than freezing parts.\n\n> I'm using your keras centernet, it's wonderful, may I ask if this is the weight you train or get from original repo?\n\nThe weights are from the original repo. I hope I find some time to update the code to tf 2.0 and add the training part. It's much more efficient and simpler to use with keras.", "706648": "A rough guess is faster than 10 FPS on a 1080 TI without TTA.", "706657": "thanks very much.😃" }, "source": "meta" }