![\"model](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F473234%2F30dfe0e6ec593a3824841290c0b5df98%2F10th_place_model_overview_v2.png?generation=1584668448561081&alt=media\")
I added the additional study at the end of this post.
\n* link to discussion about DataAugmentation\n* links to discussion and notebook about visualization of sSE Block \n* tables of my late submission results (w/o RandomErasing, w/o sSE Block, w/ common sSE Block, and w/ Chris's Magic)\n* link to the source code repository on GitHub
I was more surprised than pleased when the private leaderboard uncovered. I didn't imagine such a big shake happens.\nI still don't understand why got this place (sorry, since I had only 4 days, I was not able to do enough experiments) , and want to investigate it by late submission.
\n\nAlthough there remain some mysteries, this is my first solo gold medal. I'm so glad to share my solution as a gold place one and become Competitions Master!
\n\nLastly, congrats to all the teams got the medal and participants finished this competition! And thanks to Kaggle and Bengali.AI for hosting this competition!
\n\n
\nHere, I'd like to share my solution's summary.
Kaggle notebooks and a local machine (GTX1080ti x 1)
\n\nAt first, for tackling the unseen graphemes problem, I split train data by multi-label stratified group K-fold (regarding a grapheme as a group) and tried training models. However, I was not able to train models successfully. I suspect this is because some components are contained only in one or few grapheme(s).
\n\nMy approach is very simple － training a model using all train data (because I want to include all components in training). This way, However, has a risk of overfitting. For preventing worsening of generalization performance, I train a model using cosine annealing scheduling and do snapshot ensemble.
\n\nThere is no special pre-processing nor post-processing.
\n\nThe model is based on SE-ResNext50 and not so special, but has something unique in global pooling step. The figure below is the model overview.
\n\n
Instead of simply applying Global Average Pooling to feature map and using its result as common input of each component's head, I used sSE Block and Global Average Pooling for each component.
\n\nIf you want to know more details about sSE Block, read the original paper:
\n\n\n\nNOTE: For passing 1-channel images to the pre-trained model(3-channel), I use a simple way. See the comment bellow for more details.
\n\napplying augmentations by the following order, implemented by albumentations
\n\n[Original Image: 137x236]\n-> Padding(to 140x245) -> Rotate(rotate limit=5, p=0.8)\n-> Resize(to 128x224) -> RandomScale(scale limit=0.1, p=1.0)\n-> Padding(to 146x256) -> RandomCrop(to 128x224, p=1.0)\n-> RandomErasing(mask-ratio-min=0.02, mask-ratio-max=0.4, p=0.5)\n-> Normalize(by channel-wise mean and std of train data)\n-> [Model Input: 128x224]
\n\nAs you see, I did not use CutMix/MixUp.
\n\ncalculating softmax cross entropy for each component, and averaging them by weights (grapheme_root:vowel_diacritic:consonant_diacritic = 2 : 1 : 1)
applying the following transforms to images before feeding them into models
\n\n[Original Image: 137x236]\n-> Padding(to 140x245) -> Resize(to 128x224)\n-> Normalize(by channel-wise mean and std of train data)\n-> [Model Input: 128x224]
\n\nObtaining models(35, 75, and 105 epoch)' outputs using softmax by component, simply averaging them and applying argmax by component.
\n\nI don't still understand, but have some hypotheses.
\n\nMany participants used CutMix or MixUp. While these techniques seem effective in making combinations of components, not effective in disentangling interdependence of components in original combinations, especially about MixUp.
\n\nBecause of this, may be, simple masking method like RandomErasing is good.
\n\nI'm not sure whether or not sSE Block is the best. But simply applying global average pooling looks not good for me because I suspect each component has spacial dependence.
\n\nI suppose a kind of weighted global average pooling for each component is good.
\n\nIn previous competitions, training K-fold and K-fold averaging was the very effective way for me. But in this competition, it was very difficult for me to split K-fold because of unseen graphemes problem and rare components, I trained a model using all train data.
\n\nHere are each cycle's and snapshot ensemble's scores.
\n\n| model | Public Score (Rank) | Private Score (Rank) |\n|:-------------------:|:-------:|:--------|\n| cycle 1 (35epoch) | 0.9754 (288th) | 0.9438 (23rd) |\n| cycle 2 (70epoch) | 0.9821 (160th) | 0.9499 (14th) |\n| cycle 3 (105epoch) | 0.9843 (124th) | 0.9497 (14th) |\n| snapshot ensemble | 0.9840 (131st) | 0.9536 (10th) |
\n\nThe model cycle 3 achieved the best public score, but this slightly overfitted.\nSnapshot ensemble achieved the best private score, which is 0.0037 higher than the best single model(cycle 2).
\n\nI think snapshot ensemble is good when you want to train models using all train data.
\n\n
\nThat's all. Thank you for reading!
| model | cycle | Public Score | Private Score |\n|:-------:|:-------------------:|:-------:|:--------|\n| w/o RandomErasing | 1 (35epoch) | 0.9425 | 0.9187 |\n| 〃 | 2 (70epoch) | 0.9479 | 0.9201 |\n| 〃 | 3 (105epoch) | 0.9648 | 0.9345 |\n| 〃 | Snapshot Ensemble | 0.9643 | 0.9373 |\n| w/o sSE Module | 1 (35epoch) | 0.9746 | 0.9431 |\n| 〃 | 2 (70epoch) | 0.9824 | 0.9490 |\n| 〃 | 3 (105epoch) | 0.9836 | 0.9485 |\n| 〃 | Snapshot Ensemble | 0.9843 | 0.9517 |\n| w/ Common sSE Module | 1 (35epoch) | 0.9739 | 0.9433 |\n| 〃 | 2 (70epoch) | 0.9809 | 0.9474 |\n| 〃 | 3 (105epoch) | 0.9827 | 0.9496 |\n| 〃 | Snapshot Ensemble | 0.9832 | 0.9527 |
\n\nCompare results using Snapshot Ensemble (& Magical Post-Processing)\n\n| model | Public Score (rank) | Private Score (rank) |\n|:--------:|:----------- -:|:---------------|\n| w/o RandomErasing | 0.9643 (1210th) | 0.9373 (66th) |\n| w/o sSE Module | 0.9843 (124th) | 0.9517 (12th) |\n| w/ Common sSE Module | 0.9832 (149th) | 0.9527 (12th) |\n| final sub model (component-wise sSE Module) | 0.9840 (131st) | 0.9536 (10th) |\n| w/ Chris's Magic(-0.6,-0.6,-0.4) | 0.9848 (112th) | 0.9623 (4th) |\n| w/ Chris's Magic(-0.8,-0.8,-0.7) | 0.9828 (153rd) | 0.9653 (3rd) |
\n\nOMG! It's a really magic!
\n\nIf you want to use this magic (it is very easy to use but effective!), check this discussion: https://www.kaggle.com/c/bengaliai-cv19/discussion/136021
\n\nI've published the source code on GitHub:\nhttps://github.com/tawatawara/kaggle-bengaliai-cv19
", "messages": [ { "id": 777797, "postDate": "2020-03-18T00:14:01.563Z", "content": "I added the additional study at the end of this post.
\n* link to discussion about DataAugmentation\n* links to discussion and notebook about visualization of sSE Block \n* tables of my late submission results (w/o RandomErasing, w/o sSE Block, w/ common sSE Block, and w/ Chris's Magic)\n* link to the source code repository on GitHub
I was more surprised than pleased when the private leaderboard uncovered. I didn't imagine such a big shake happens.\nI still don't understand why got this place (sorry, since I had only 4 days, I was not able to do enough experiments) , and want to investigate it by late submission.
\n\nAlthough there remain some mysteries, this is my first solo gold medal. I'm so glad to share my solution as a gold place one and become Competitions Master!
\n\nLastly, congrats to all the teams got the medal and participants finished this competition! And thanks to Kaggle and Bengali.AI for hosting this competition!
\n\n
\nHere, I'd like to share my solution's summary.
Kaggle notebooks and a local machine (GTX1080ti x 1)
\n\nAt first, for tackling the unseen graphemes problem, I split train data by multi-label stratified group K-fold (regarding a grapheme as a group) and tried training models. However, I was not able to train models successfully. I suspect this is because some components are contained only in one or few grapheme(s).
\n\nMy approach is very simple － training a model using all train data (because I want to include all components in training). This way, However, has a risk of overfitting. For preventing worsening of generalization performance, I train a model using cosine annealing scheduling and do snapshot ensemble.
\n\nThere is no special pre-processing nor post-processing.
\n\nThe model is based on SE-ResNext50 and not so special, but has something unique in global pooling step. The figure below is the model overview.
\n\n
Instead of simply applying Global Average Pooling to feature map and using its result as common input of each component's head, I used sSE Block and Global Average Pooling for each component.
\n\nIf you want to know more details about sSE Block, read the original paper:
\n\n\n\nNOTE: For passing 1-channel images to the pre-trained model(3-channel), I use a simple way. See the comment bellow for more details.
\n\napplying augmentations by the following order, implemented by albumentations
\n\n[Original Image: 137x236]\n-> Padding(to 140x245) -> Rotate(rotate limit=5, p=0.8)\n-> Resize(to 128x224) -> RandomScale(scale limit=0.1, p=1.0)\n-> Padding(to 146x256) -> RandomCrop(to 128x224, p=1.0)\n-> RandomErasing(mask-ratio-min=0.02, mask-ratio-max=0.4, p=0.5)\n-> Normalize(by channel-wise mean and std of train data)\n-> [Model Input: 128x224]
\n\nAs you see, I did not use CutMix/MixUp.
\n\ncalculating softmax cross entropy for each component, and averaging them by weights (grapheme_root:vowel_diacritic:consonant_diacritic = 2 : 1 : 1)
applying the following transforms to images before feeding them into models
\n\n[Original Image: 137x236]\n-> Padding(to 140x245) -> Resize(to 128x224)\n-> Normalize(by channel-wise mean and std of train data)\n-> [Model Input: 128x224]
\n\nObtaining models(35, 75, and 105 epoch)' outputs using softmax by component, simply averaging them and applying argmax by component.
\n\nI don't still understand, but have some hypotheses.
\n\nMany participants used CutMix or MixUp. While these techniques seem effective in making combinations of components, not effective in disentangling interdependence of components in original combinations, especially about MixUp.
\n\nBecause of this, may be, simple masking method like RandomErasing is good.
\n\nI'm not sure whether or not sSE Block is the best. But simply applying global average pooling looks not good for me because I suspect each component has spacial dependence.
\n\nI suppose a kind of weighted global average pooling for each component is good.
\n\nIn previous competitions, training K-fold and K-fold averaging was the very effective way for me. But in this competition, it was very difficult for me to split K-fold because of unseen graphemes problem and rare components, I trained a model using all train data.
\n\nHere are each cycle's and snapshot ensemble's scores.
\n\n| model | Public Score (Rank) | Private Score (Rank) |\n|:-------------------:|:-------:|:--------|\n| cycle 1 (35epoch) | 0.9754 (288th) | 0.9438 (23rd) |\n| cycle 2 (70epoch) | 0.9821 (160th) | 0.9499 (14th) |\n| cycle 3 (105epoch) | 0.9843 (124th) | 0.9497 (14th) |\n| snapshot ensemble | 0.9840 (131st) | 0.9536 (10th) |
\n\nThe model cycle 3 achieved the best public score, but this slightly overfitted.\nSnapshot ensemble achieved the best private score, which is 0.0037 higher than the best single model(cycle 2).
\n\nI think snapshot ensemble is good when you want to train models using all train data.
\n\n
\nThat's all. Thank you for reading!
| model | cycle | Public Score | Private Score |\n|:-------:|:-------------------:|:-------:|:--------|\n| w/o RandomErasing | 1 (35epoch) | 0.9425 | 0.9187 |\n| 〃 | 2 (70epoch) | 0.9479 | 0.9201 |\n| 〃 | 3 (105epoch) | 0.9648 | 0.9345 |\n| 〃 | Snapshot Ensemble | 0.9643 | 0.9373 |\n| w/o sSE Module | 1 (35epoch) | 0.9746 | 0.9431 |\n| 〃 | 2 (70epoch) | 0.9824 | 0.9490 |\n| 〃 | 3 (105epoch) | 0.9836 | 0.9485 |\n| 〃 | Snapshot Ensemble | 0.9843 | 0.9517 |\n| w/ Common sSE Module | 1 (35epoch) | 0.9739 | 0.9433 |\n| 〃 | 2 (70epoch) | 0.9809 | 0.9474 |\n| 〃 | 3 (105epoch) | 0.9827 | 0.9496 |\n| 〃 | Snapshot Ensemble | 0.9832 | 0.9527 |
\n\nCompare results using Snapshot Ensemble (& Magical Post-Processing)\n\n| model | Public Score (rank) | Private Score (rank) |\n|:--------:|:----------- -:|:---------------|\n| w/o RandomErasing | 0.9643 (1210th) | 0.9373 (66th) |\n| w/o sSE Module | 0.9843 (124th) | 0.9517 (12th) |\n| w/ Common sSE Module | 0.9832 (149th) | 0.9527 (12th) |\n| final sub model (component-wise sSE Module) | 0.9840 (131st) | 0.9536 (10th) |\n| w/ Chris's Magic(-0.6,-0.6,-0.4) | 0.9848 (112th) | 0.9623 (4th) |\n| w/ Chris's Magic(-0.8,-0.8,-0.7) | 0.9828 (153rd) | 0.9653 (3rd) |
\n\nOMG! It's a really magic!
\n\nIf you want to use this magic (it is very easy to use but effective!), check this discussion: https://www.kaggle.com/c/bengaliai-cv19/discussion/136021
\n\nI've published the source code on GitHub:\nhttps://github.com/tawatawara/kaggle-bengaliai-cv19
", "rawMarkdown": "## UPDATED\nI added the additional study at the end of this post. \n* link to discussion about DataAugmentation\n* links to discussion and notebook about visualization of sSE Block \n* tables of my late submission results (w/o RandomErasing, w/o sSE Block, w/ common sSE Block, and w/ **_Chris's Magic_**)\n* link to the source code repository on GitHub\n\n----\n\nI was more surprised than pleased when the private leaderboard uncovered. I didn't imagine such a big shake happens.\nI still don't understand why got this place (sorry, since I had only 4 days, I was not able to do enough experiments) , and want to investigate it by late submission.\n\nAlthough there remain some mysteries, this is my first solo gold medal. I'm so glad to share my solution as a gold place one and become Competitions Master!\n\nLastly, congrats to all the teams got the medal and participants finished this competition! And thanks to Kaggle and Bengali.AI for hosting this competition!\n\nThanks for sharing and congratulations on your solo gold!! Are you passing in the 3-channel image to your model? In your diagram, it's 1-channel; if it's 1-channel then there has to be a conv-layer to convert into 3-channel before passing in to pretrained serex50
", "rawMarkdown": "Thanks for sharing and congratulations on your solo gold!! Are you passing in the 3-channel image to your model? In your diagram, it's 1-channel; if it's 1-channel then there has to be a conv-layer to convert into 3-channel before passing in to pretrained serex50", "votes": 1, "replies": [ { "id": 781853, "postDate": "2020-03-21T17:46:26.883Z", "content": "Thanks!
\n\nNo, I'm passing 1-channel images to SE-ResNeXt50, but this is the pre-trained model(3-channel) on ImageNet.\nI do this in a naive way.
\n\nThe first conv-layer of SE-ResNeXt50 is Conv2D(in_channels=3, out_channels=64, kernel_size=(7,7), stride=(2,2), pad=(3,3)), of which weight has the shape of (64, 3, 7, 7) as bellow.
\n(NOTE: I think this shape depends on your framework. I'm using Chainer and ChainerCV.)
python\n>>> import numpy as np\n>>> import chainercv\n>>> model = chainercv.links.SEResNet50(pretrained_model=\"imagenet\")\n>>> model.conv1.conv.W.data.shape\n(64, 3, 7, 7)\n
I simply average it along axis=1 which is corresponded to in_channels, then add a new-axis to this average weight.
python\n>>> w_mean = model.conv1.conv.W.data.mean(axis=1)\n>>> w_mean.shape\n(64, 7, 7)\n>>> w_mean = w_mean[:, np.newaxis, :, :]\n>>> w_mean.shape\n(64, 1, 7, 7)\n
Finally, I use it as a weight of the first conv-layer of SE-ResNeXt50.
\n\n\n>>> model.conv1.conv.W.data = w_mean\n>>> model.conv1.conv.W.data.shape\n(64, 1, 7, 7)\n>>> # # passing examples # #\n>>> x = np.random.rand(4, 1, 128, 224).astype(\"f\")\n>>> x.shape # shape=(batch_size, channel, height, width)\n(4, 1, 128, 224)\n>>> y = model(x)\n>>> y.shape # shape=(bs, output_class_num)\n(4, 1000)\n
I once pushed publish topic by mistake, sorry😅 .
Thanks!! I'm currently trying to replicate your results with pytorch and with 3-channel image. I'll update my results once I'm done training. Just to be clear: you save model checkpoints at epoch 35, 70 and 105 epochs, right?
", "rawMarkdown": "Thanks!! I'm currently trying to replicate your results with pytorch and with 3-channel image. I'll update my results once I'm done training. Just to be clear: you save model checkpoints at epoch 35, 70 and 105 epochs, right?", "votes": 1 }, { "id": 781976, "postDate": "2020-03-21T20:05:52.110Z", "content": "Thank you for trying to reproduce! Yes, I saved the model at epoch 35, 70, and 105.
\n\nFor your reference, I show you my final model's training curve.
\n\n![\"training](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F473234%2Fc1a76e0dbf058bc81471e746027c8359%2FBengaliAI_CV2019_20200322045207.png?generation=1584820441866694&alt=media\")
Regards.
", "rawMarkdown": "Thank you for trying to reproduce! Yes, I saved the model at epoch 35, 70, and 105.\n\nFor your reference, I show you my final model's training curve.\n\n\n\n* blue line (left axis) : training loss\n * It is jaggy because represents the loss of last iteration at each epoch.\n* pale red line (right axis): learning rate\n * cosine anealing scheduling, lr-max=1.5e-02, lr-min=0.0\n* There is no validation loss **because of training by all train data**.\n\nRegards.", "votes": 2 } ] }, { "id": 778407, "postDate": "2020-03-18T12:16:08.607Z", "content": "Nice approach
", "rawMarkdown": "Nice approach ", "votes": 1, "replies": [ { "id": 778459, "postDate": "2020-03-18T13:11:56.327Z", "content": "Thanks!
", "rawMarkdown": "Thanks!" } ] }, { "id": 777954, "postDate": "2020-03-18T03:23:20.093Z", "content": "I guess the biggest factor is training on the whole dataset? I tried training on different folds and ensemble but not sure if it does the same thing as training on the whole dataset. I encountered a big shake-down.
", "rawMarkdown": "I guess the biggest factor is training on the whole dataset? I tried training on different folds and ensemble but not sure if it does the same thing as training on the whole dataset. I encountered a big shake-down.", "votes": 1, "replies": [ { "id": 778442, "postDate": "2020-03-18T12:56:29.510Z", "content": "We tried a snapshot ensemble by training on the entire dataset, using the number of validation epochs as a guide for early stopping. We still shook down. So I think it has more to do with sSEBlock and RandomErasing. @ttahara , congratulations on your gold!
", "rawMarkdown": "We tried a snapshot ensemble by training on the entire dataset, using the number of validation epochs as a guide for early stopping. We still shook down. So I think it has more to do with sSEBlock and RandomErasing. @ttahara , congratulations on your gold!", "votes": 2 }, { "id": 778456, "postDate": "2020-03-18T13:10:49.823Z", "content": "Thanks @returnofsputnik !
\n\nI have a similar thought as yours because my each cycle model got silver or gold place.\nIn my intuition, using RandomErasing but not using CutMix/MixUp is more important than sSE Block.
", "rawMarkdown": "Thanks @returnofsputnik !\n\nI have a similar thought as yours because my each cycle model got silver or gold place.\nIn my intuition, using RandomErasing **but not using CutMix/MixUp** is more important than sSE Block. \n", "votes": 1 }, { "id": 778464, "postDate": "2020-03-18T13:18:00.850Z", "content": "@ttahara Is there any differnce between RandomErasing and Cutout augmentations? Thank you
I have the opposite intuition. wouldn't erase/cutout force the model to remember combinations / associate different components?
", "rawMarkdown": "I have the opposite intuition. wouldn't erase/cutout force the model to remember combinations / associate different components?", "votes": 1 }, { "id": 778925, "postDate": "2020-03-18T21:25:14.580Z", "content": "\n\nAs far as I know, these two have the following differnces.( sorry if there is any mistake )
\n\n| property ＼ Method | Cutout | RandomErasing |\n|:------------|:--------------------------|:---------------|\n| mask size | fixed (hyper param) | random (decided from ranges of hyper param)|\n| mask aspect ratio | fixed | random (decided from ranges of hyper param) |\n| mask pixel value | fixed(mean value) | random (or fixed value) |\n| mask location | random (if mask crosses edges, the overflown is cut ) | random (decided to not cross edges) |\n| mask num| 1 or more than 1 (hyperparam) | 1 |
\n\nif you want to know exact information, I recommend you to read the original papers and codes:
\n\nCutout\nRandomErasing\n@yl1202 Thanks! that's a very useful comment for me.
\n\nI think your intuition is right if RandomErasing/CutOut are always applied, but practically, these are stochastically applied (e.g. p=0.5).
", "rawMarkdown": "@yl1202 Thanks! that's a very useful comment for me.\n\nI think your intuition is right if RandomErasing/CutOut are **always** applied, but practically, these are **stochastically** applied (e.g. p=0.5).", "votes": 1 } ] }, { "id": 777911, "postDate": "2020-03-18T02:46:20.870Z", "content": "👍 👍 👍
", "rawMarkdown": "👍 👍 👍 ", "votes": 1, "replies": [ { "id": 778458, "postDate": "2020-03-18T13:11:40.240Z", "content": "Thanks!
", "rawMarkdown": "Thanks!" } ] }, { "id": 777891, "postDate": "2020-03-18T02:29:23.500Z", "content": "Hi, thank you for sharing and congrats on 10th place.\nI'm amazed you achieved that score without postprocessing such as https://www.kaggle.com/c/bengaliai-cv19/discussion/136021
", "rawMarkdown": "Hi, thank you for sharing and congrats on 10th place.\nI'm amazed you achieved that score without postprocessing such as https://www.kaggle.com/c/bengaliai-cv19/discussion/136021", "votes": 1, "replies": [ { "id": 777988, "postDate": "2020-03-18T04:03:55.437Z", "content": "Thanks! I didn't think of that post-processing. I'll try it.
", "rawMarkdown": "Thanks! I didn't think of that post-processing. I'll try it.", "votes": 1 } ] }, { "id": 777828, "postDate": "2020-03-18T01:26:43.177Z", "content": "Thank you for sharing and congratulations!!
", "rawMarkdown": "Thank you for sharing and congratulations!!", "votes": 1, "replies": [ { "id": 777920, "postDate": "2020-03-18T02:51:09.950Z", "content": "Thanks!
", "rawMarkdown": "Thanks!" } ] }, { "id": 779039, "postDate": "2020-03-19T01:06:49.547Z", "content": "Congrats on your 10th place and awesome solo gold! \nOne of the reasons why your model performs so well in Private may be your validation scheme. You used only unseen graphemes for validation until the middle stage of this competition (according to your tweet). I think it made your base model and approach robust to unseen graphemes, even though it might not be so apparent in your final model architecture and settings.
", "rawMarkdown": "Congrats on your 10th place and awesome solo gold! \nOne of the reasons why your model performs so well in Private may be your validation scheme. You used only unseen graphemes for validation until the middle stage of this competition (according to your tweet). I think it made your base model and approach robust to unseen graphemes, even though it might not be so apparent in your final model architecture and settings.", "votes": 2, "replies": [ { "id": 779531, "postDate": "2020-03-19T12:54:32.750Z", "content": "Thanks and congrats on your 6th place!
\n\nYes, you are right. For the first two days, I validated models by validation set split in that manner and decided several settings such as parameters of data augmentation.
\n\nBut validation loss would not decrease more at an early epoch (5 ~ 7). I was very troubled😅
", "rawMarkdown": "Thanks and congrats on your 6th place!\n\nYes, you are right. For the first two days, I validated models by validation set split in that manner and decided several settings such as parameters of data augmentation.\n\nBut validation loss would not decrease more at an early epoch (5 ~ 7). I was very troubled😅 ", "votes": 2 } ] }, { "id": 777888, "postDate": "2020-03-18T02:27:29.290Z", "content": "Really interesting, you didn't do any optimization on unseen graphemes but your models did it for you!\nThat's impressive!
", "rawMarkdown": "Really interesting, you didn't do any optimization on unseen graphemes but your models did it for you!\nThat's impressive!", "votes": 2, "replies": [ { "id": 777932, "postDate": "2020-03-18T03:00:39.387Z", "content": "Thanks and congrats on 8th place! I'm really surprised at this result, too.
\n\nAlthough I still don't understand, I'm happy to get this place by very simple solution.
", "rawMarkdown": "Thanks and congrats on 8th place! I'm really surprised at this result, too.\n\nAlthough I still don't understand, I'm happy to get this place by very simple solution.", "votes": 1 } ] }, { "id": 795959, "postDate": "2020-04-03T06:41:11.283Z", "content": "Hi @Tawara
\n\nCongrats on 10th place!
\n\nCould I have one question?\nWhy did you pad image to different image size then do Rotate/ RandomCrop?
\n\nLooking forward to your reply.
", "rawMarkdown": "Hi @Tawara\n\nCongrats on 10th place!\n\nCould I have one question?\nWhy did you pad image to different image size then do Rotate/ RandomCrop?\n\nLooking forward to your reply." }, { "id": 789297, "postDate": "2020-03-28T14:33:05.923Z", "content": "I updated the result of visualization and late submissions. The source code is now available on GitHub.
\nI think Chris's magic is a really magic😂
Thanks.
", "rawMarkdown": "I updated the result of visualization and late submissions. The source code is now available on GitHub. \nI think Chris's magic is a really magic😂\n\nThanks." }, { "id": 780080, "postDate": "2020-03-20T00:22:56.677Z", "content": "One question, when predict how did you process the images? Directly resize to 128x224?
", "rawMarkdown": "One question, when predict how did you process the images? Directly resize to 128x224?", "replies": [ { "id": 780090, "postDate": "2020-03-20T00:59:46.430Z", "content": "Sorry, that is represented by Padding + Resize in the overview figure but somewhat ambiguous.
I applied padding original images[137x236] to 140x245, then resized them into 128x224 (and normalized them).
\n\nI added this information and corrected the figure. Thanks!
", "rawMarkdown": "Sorry, that is represented by `Padding + Resize` in the overview figure but somewhat ambiguous.\n\nI applied padding original images[137x236] to 140x245, then resized them into 128x224 (and normalized them).\n\nI added this information and corrected the figure. Thanks!", "votes": 1 } ] }, { "id": 779414, "postDate": "2020-03-19T10:19:03.717Z", "content": "Congrats for solo Gold and becoming competition master. From your solution I learned many things that I didn't know. thanks for sharing.
", "rawMarkdown": "Congrats for solo Gold and becoming competition master. From your solution I learned many things that I didn't know. thanks for sharing.", "replies": [ { "id": 779538, "postDate": "2020-03-19T13:04:59.473Z", "content": "Thanks! I'm glad that my solution gives you many impressions.
", "rawMarkdown": "Thanks! I'm glad that my solution gives you many impressions." } ] }, { "id": 781646, "postDate": "2020-03-21T14:25:19.710Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 781833, "postDate": "2020-03-21T17:19:28.377Z", "content": "Thanks!
", "rawMarkdown": "Thanks!" } ] } ], "comments": [ { "id": 781819, "author_name": "Bibek", "author_url": "", "post_date": "2020-03-21T17:05:19.840000", "content": "Thanks for sharing and congratulations on your solo gold!! Are you passing in the 3-channel image to your model? In your diagram, it's 1-channel; if it's 1-channel then there has to be a conv-layer to convert into 3-channel before passing in to pretrained serex50
", "votes": 1, "replies": [ { "id": 781853, "author_name": "Tawara", "author_url": "", "post_date": "2020-03-21T17:46:26.883000", "content": "Thanks!
\n\nNo, I'm passing 1-channel images to SE-ResNeXt50, but this is the pre-trained model(3-channel) on ImageNet.\nI do this in a naive way.
\n\nThe first conv-layer of SE-ResNeXt50 is Conv2D(in_channels=3, out_channels=64, kernel_size=(7,7), stride=(2,2), pad=(3,3)), of which weight has the shape of (64, 3, 7, 7) as bellow.
\n(NOTE: I think this shape depends on your framework. I'm using Chainer and ChainerCV.)
python\n>>> import numpy as np\n>>> import chainercv\n>>> model = chainercv.links.SEResNet50(pretrained_model=\"imagenet\")\n>>> model.conv1.conv.W.data.shape\n(64, 3, 7, 7)\n
I simply average it along axis=1 which is corresponded to in_channels, then add a new-axis to this average weight.
python\n>>> w_mean = model.conv1.conv.W.data.mean(axis=1)\n>>> w_mean.shape\n(64, 7, 7)\n>>> w_mean = w_mean[:, np.newaxis, :, :]\n>>> w_mean.shape\n(64, 1, 7, 7)\n
Finally, I use it as a weight of the first conv-layer of SE-ResNeXt50.
\n\n\n>>> model.conv1.conv.W.data = w_mean\n>>> model.conv1.conv.W.data.shape\n(64, 1, 7, 7)\n>>> # # passing examples # #\n>>> x = np.random.rand(4, 1, 128, 224).astype(\"f\")\n>>> x.shape # shape=(batch_size, channel, height, width)\n(4, 1, 128, 224)\n>>> y = model(x)\n>>> y.shape # shape=(bs, output_class_num)\n(4, 1000)\n
I once pushed publish topic by mistake, sorry😅 .
Thanks!! I'm currently trying to replicate your results with pytorch and with 3-channel image. I'll update my results once I'm done training. Just to be clear: you save model checkpoints at epoch 35, 70 and 105 epochs, right?
", "votes": 1, "replies": [] }, { "id": 781976, "author_name": "Tawara", "author_url": "", "post_date": "2020-03-21T20:05:52.110000", "content": "Thank you for trying to reproduce! Yes, I saved the model at epoch 35, 70, and 105.
\n\nFor your reference, I show you my final model's training curve.
\n\n
Regards.
", "votes": 2, "replies": [] } ] }, { "id": 778407, "author_name": "Kavitha Kavi", "author_url": "", "post_date": "2020-03-18T12:16:08.607000", "content": "Nice approach
", "votes": 1, "replies": [ { "id": 778459, "author_name": "Tawara", "author_url": "", "post_date": "2020-03-18T13:11:56.327000", "content": "Thanks!
", "votes": 0, "replies": [] } ] }, { "id": 777954, "author_name": "Gold Retriever", "author_url": "", "post_date": "2020-03-18T03:23:20.093000", "content": "I guess the biggest factor is training on the whole dataset? I tried training on different folds and ensemble but not sure if it does the same thing as training on the whole dataset. I encountered a big shake-down.
", "votes": 1, "replies": [ { "id": 778442, "author_name": "CoreyJamesLevinson", "author_url": "", "post_date": "2020-03-18T12:56:29.510000", "content": "We tried a snapshot ensemble by training on the entire dataset, using the number of validation epochs as a guide for early stopping. We still shook down. So I think it has more to do with sSEBlock and RandomErasing. @ttahara , congratulations on your gold!
", "votes": 2, "replies": [] }, { "id": 778456, "author_name": "Tawara", "author_url": "", "post_date": "2020-03-18T13:10:49.823000", "content": "Thanks @returnofsputnik !
\n\nI have a similar thought as yours because my each cycle model got silver or gold place.\nIn my intuition, using RandomErasing but not using CutMix/MixUp is more important than sSE Block.
", "votes": 1, "replies": [] }, { "id": 778464, "author_name": "CoreyJamesLevinson", "author_url": "", "post_date": "2020-03-18T13:18:00.850000", "content": "@ttahara Is there any differnce between RandomErasing and Cutout augmentations? Thank you
I have the opposite intuition. wouldn't erase/cutout force the model to remember combinations / associate different components?
", "votes": 1, "replies": [] }, { "id": 778925, "author_name": "Tawara", "author_url": "", "post_date": "2020-03-18T21:25:14.580000", "content": "\n\nAs far as I know, these two have the following differnces.( sorry if there is any mistake )
\n\n| property ＼ Method | Cutout | RandomErasing |\n|:------------|:--------------------------|:---------------|\n| mask size | fixed (hyper param) | random (decided from ranges of hyper param)|\n| mask aspect ratio | fixed | random (decided from ranges of hyper param) |\n| mask pixel value | fixed(mean value) | random (or fixed value) |\n| mask location | random (if mask crosses edges, the overflown is cut ) | random (decided to not cross edges) |\n| mask num| 1 or more than 1 (hyperparam) | 1 |
\n\nif you want to know exact information, I recommend you to read the original papers and codes:
\n\nCutout\nRandomErasing\n@yl1202 Thanks! that's a very useful comment for me.
\n\nI think your intuition is right if RandomErasing/CutOut are always applied, but practically, these are stochastically applied (e.g. p=0.5).
", "votes": 1, "replies": [] } ] }, { "id": 777911, "author_name": "Tian", "author_url": "", "post_date": "2020-03-18T02:46:20.870000", "content": "👍 👍 👍
", "votes": 1, "replies": [ { "id": 778458, "author_name": "Tawara", "author_url": "", "post_date": "2020-03-18T13:11:40.240000", "content": "Thanks!
", "votes": 0, "replies": [] } ] }, { "id": 777891, "author_name": "Appian", "author_url": "", "post_date": "2020-03-18T02:29:23.500000", "content": "Hi, thank you for sharing and congrats on 10th place.\nI'm amazed you achieved that score without postprocessing such as https://www.kaggle.com/c/bengaliai-cv19/discussion/136021
", "votes": 1, "replies": [ { "id": 777988, "author_name": "Tawara", "author_url": "", "post_date": "2020-03-18T04:03:55.437000", "content": "Thanks! I didn't think of that post-processing. I'll try it.
", "votes": 1, "replies": [] } ] }, { "id": 777828, "author_name": "mocobt", "author_url": "", "post_date": "2020-03-18T01:26:43.177000", "content": "Thank you for sharing and congratulations!!
", "votes": 1, "replies": [ { "id": 777920, "author_name": "Tawara", "author_url": "", "post_date": "2020-03-18T02:51:09.950000", "content": "Thanks!
", "votes": 0, "replies": [] } ] }, { "id": 779039, "author_name": "YuyaYamamoto", "author_url": "", "post_date": "2020-03-19T01:06:49.547000", "content": "Congrats on your 10th place and awesome solo gold! \nOne of the reasons why your model performs so well in Private may be your validation scheme. You used only unseen graphemes for validation until the middle stage of this competition (according to your tweet). I think it made your base model and approach robust to unseen graphemes, even though it might not be so apparent in your final model architecture and settings.
", "votes": 2, "replies": [ { "id": 779531, "author_name": "Tawara", "author_url": "", "post_date": "2020-03-19T12:54:32.750000", "content": "Thanks and congrats on your 6th place!
\n\nYes, you are right. For the first two days, I validated models by validation set split in that manner and decided several settings such as parameters of data augmentation.
\n\nBut validation loss would not decrease more at an early epoch (5 ~ 7). I was very troubled😅
", "votes": 2, "replies": [] } ] }, { "id": 777888, "author_name": "Qishen Ha", "author_url": "", "post_date": "2020-03-18T02:27:29.290000", "content": "Really interesting, you didn't do any optimization on unseen graphemes but your models did it for you!\nThat's impressive!
", "votes": 2, "replies": [ { "id": 777932, "author_name": "Tawara", "author_url": "", "post_date": "2020-03-18T03:00:39.387000", "content": "Thanks and congrats on 8th place! I'm really surprised at this result, too.
\n\nAlthough I still don't understand, I'm happy to get this place by very simple solution.
", "votes": 1, "replies": [] } ] }, { "id": 795959, "author_name": "YS", "author_url": "", "post_date": "2020-04-03T06:41:11.283000", "content": "Hi @Tawara
\n\nCongrats on 10th place!
\n\nCould I have one question?\nWhy did you pad image to different image size then do Rotate/ RandomCrop?
\n\nLooking forward to your reply.
", "votes": 0, "replies": [] }, { "id": 789297, "author_name": "Tawara", "author_url": "", "post_date": "2020-03-28T14:33:05.923000", "content": "I updated the result of visualization and late submissions. The source code is now available on GitHub.
\nI think Chris's magic is a really magic😂
Thanks.
", "votes": 0, "replies": [] }, { "id": 780080, "author_name": "Strideradu", "author_url": "", "post_date": "2020-03-20T00:22:56.677000", "content": "One question, when predict how did you process the images? Directly resize to 128x224?
", "votes": 0, "replies": [ { "id": 780090, "author_name": "Tawara", "author_url": "", "post_date": "2020-03-20T00:59:46.430000", "content": "Sorry, that is represented by Padding + Resize in the overview figure but somewhat ambiguous.
I applied padding original images[137x236] to 140x245, then resized them into 128x224 (and normalized them).
\n\nI added this information and corrected the figure. Thanks!
", "votes": 1, "replies": [] } ] }, { "id": 779414, "author_name": "Raghawendra Singh", "author_url": "", "post_date": "2020-03-19T10:19:03.717000", "content": "Congrats for solo Gold and becoming competition master. From your solution I learned many things that I didn't know. thanks for sharing.
", "votes": 0, "replies": [ { "id": 779538, "author_name": "Tawara", "author_url": "", "post_date": "2020-03-19T13:04:59.473000", "content": "Thanks! I'm glad that my solution gives you many impressions.
", "votes": 0, "replies": [] } ] }, { "id": 781646, "author_name": "", "author_url": "", "post_date": "2020-03-21T14:25:19.710000", "content": "", "votes": 0, "replies": [ { "id": 781833, "author_name": "Tawara", "author_url": "", "post_date": "2020-03-21T17:19:28.377000", "content": "Thanks!
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "777797": "## UPDATED\nI added the additional study at the end of this post. \n* link to discussion about DataAugmentation\n* links to discussion and notebook about visualization of sSE Block \n* tables of my late submission results (w/o RandomErasing, w/o sSE Block, w/ common sSE Block, and w/ **_Chris's Magic_**)\n* link to the source code repository on GitHub\n\n----\n\nI was more surprised than pleased when the private leaderboard uncovered. I didn't imagine such a big shake happens.\nI still don't understand why got this place (sorry, since I had only 4 days, I was not able to do enough experiments) , and want to investigate it by late submission.\n\nAlthough there remain some mysteries, this is my first solo gold medal. I'm so glad to share my solution as a gold place one and become Competitions Master!\n\nLastly, congrats to all the teams got the medal and participants finished this competition! And thanks to Kaggle and Bengali.AI for hosting this competition!\n\n