![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1424766%2F3e968cbc82806b9211bf7609b662a18a%2Fbengali%20arch.001.jpeg?generation=1584456769762003&alt=media\")
Thanks at Bengali.ai and kaggle for hosting this interesting competition as it was more to it than simple ensembling of computer vision models. Thanks to the authors and contributors of pytorch, pytorch-lightning, apex and pytorchcv for making my life easier.
\n\nAcknowledgements & notes on GM
\n\nI am humbled to finally become competition grandmaster. I learned a tremendous amount of tricks in the recent two years and it would not have been able if it wasn’t for all this generous sharing of top solutions and extremely talented teammates I was lucky to have along the way.
\n\nMy solution is a simple bag (different seeds) of the same model seresnext50 with custom head. Major boost in LB score came from redesign of consonant diacritic target. Minor LB improvement from postprocessing tweaks such as thresholding and finding closest train examples.
\n\nI did not resize the image, the only thing I did was normalize each image by its mean and std, since I experienced a good regularization from that in previous competitions.
\n\nBackbone\nI used a plain seresnext50 but adjusted the very first layer to replace resizing the image and account for single channel. I did that by changing input units and reducing stride from (2,2) to (1,2).
\n\n```\nfrom pytorchcv.model_provider import get_model as ptcv_get_model
\n\nbackbone = ptcv_get_model('seresnext50_32x4d', pretrained=True)\nbackbone = backbone.features\nbackbone.init_block.conv = nn.Conv2d(1, 64, kernel_size=(7, 7), stride=(1, 2))\n```
\n\nneck
\n\nsame as in maciejsypetkowski solution https://www.kaggle.com/c/recursion-cellular-image-classification/discussion/110543
\n\nheads
\n\nSo in total my used architecture looks like this:
\n\n
Training
\n\nLosses
\n\ngrapheme loss: ArcCos + CrossEntropy
total loss = root loss + consonant loss + vowel loss + grapheme loss
Now to the key aspect of my solution. I realised all models perform poorly on consonant diacritic 3 (I abbreviate with cons3 in the following) and especially confuse it with cons4. And with poorly I mean game changing poorly: 89% recall compared to 99% for the other consonant classes. At first I just fought the symptoms (instead of the root cause) as several other teams did, e.g. by transferring low confident predicts from high frequent class cons4 to low frequent class cons3 and hence leveraging the definition of macro recall metric and that gave a good LB boost (0.9900 -> 0.9915). That might have been enough for gold but I wanted to improve my chances. So I wanted to understand why the model did so poor on this one and read through discussions, wikipedia and other domain knowledge related things. Luckily, I found 2 things:\n- Wikipedia: As the last member of a conjunct, য jô appears as a wavy vertical line (called যফলা jôfôla) to the right of the previous member: ক্য \"kyô\" খ্য \"khyô\" গ্য \"gyô\" ঘ্য \"ghyô\" etc. In some fonts, certain conjuncts with যফলা jôfôla appear using special fused forms: দ্য \"dyô\" ন্য \"nyô\" শ্য \"shyô\" ষ্য \"ṣyô\" স্য \"syô\" হ্য \"hyô\".
\n\nSo I realized that not only 3,4 and 6 have a jô component which might have the special jôfôla form but also that the special forms are in train for only cons4 but not for cons3, and thats why the models are so bad on cons3. My solution was the following. I recoded the cons classes into a multilabel classification problem using the following function:
\n\n```\ncons_components = ['ঁ', 'য', 'র','্']
\n\ndef is_sub(sub, lst):\n ln = len(sub)\n return any(lst[i: i + ln] == sub for i in range(len(lst) - ln + 1))
\n\ndef label2label_v2(label):\n res = np.zeros((6,),dtype=int)\n components = cons2components[label]\n for i,item in enumerate(cons_components):\n if item in components:\n res[i] = 1\n if is_sub(['্', 'য'], components):\n res[4] = 1\n if is_sub(['্', 'র'], components):\n res[5] = 1\n return res
\n\n```
\n\nThat function re-codes the 7 consonant class labels into multilabel 6 dimensional vectors
\n\n\n0 -> [0, 0, 0, 0, 0, 0]\n1 -> [1, 0, 0, 0, 0, 0]\n2 -> [0, 0, 1, 1, 0, 0]\n3 -> [0, 1, 1, 1, 1, 0]\n4 -> [0, 1, 0, 1, 1, 0]\n5 -> [0, 0, 1, 1, 0, 1]\n6 -> [0, 1, 1, 1, 1, 1]\n
That schema not only enabled to learn special forms of cons3 from cons4 but also enables to tune via thresholding (i.e. when to set a continuous prediction to 0 or 1) how the model separates the 6 classes. E.g. by setting a very low threshold on the 2nd logit more predictions would be set to 1 and hence more predictions would move from cons4 to cons3 as these two classes only differ in logit2. Purely changing the target for consonant diacritic in this way improved Public LB from 0.990 to 0.993 !
\n\nI tempered around with binarization and metric learning thresholds and purely checked LB hence I was aware of a high risk to potentially overfit to LB. So I chose my best sub and a failsafe sub that I felt was the least tempered with but still doing ok. (Both were gold on private at the end)
\n\nCheers.
", "messages": [ { "id": 776709, "postDate": "2020-03-17T15:27:08.870Z", "content": "Thanks at Bengali.ai and kaggle for hosting this interesting competition as it was more to it than simple ensembling of computer vision models. Thanks to the authors and contributors of pytorch, pytorch-lightning, apex and pytorchcv for making my life easier.
\n\nAcknowledgements & notes on GM
\n\nI am humbled to finally become competition grandmaster. I learned a tremendous amount of tricks in the recent two years and it would not have been able if it wasn’t for all this generous sharing of top solutions and extremely talented teammates I was lucky to have along the way.
\n\nMy solution is a simple bag (different seeds) of the same model seresnext50 with custom head. Major boost in LB score came from redesign of consonant diacritic target. Minor LB improvement from postprocessing tweaks such as thresholding and finding closest train examples.
\n\nI did not resize the image, the only thing I did was normalize each image by its mean and std, since I experienced a good regularization from that in previous competitions.
\n\nBackbone\nI used a plain seresnext50 but adjusted the very first layer to replace resizing the image and account for single channel. I did that by changing input units and reducing stride from (2,2) to (1,2).
\n\n```\nfrom pytorchcv.model_provider import get_model as ptcv_get_model
\n\nbackbone = ptcv_get_model('seresnext50_32x4d', pretrained=True)\nbackbone = backbone.features\nbackbone.init_block.conv = nn.Conv2d(1, 64, kernel_size=(7, 7), stride=(1, 2))\n```
\n\nneck
\n\nsame as in maciejsypetkowski solution https://www.kaggle.com/c/recursion-cellular-image-classification/discussion/110543
\n\nheads
\n\nSo in total my used architecture looks like this:
\n\n
Training
\n\nLosses
\n\ngrapheme loss: ArcCos + CrossEntropy
total loss = root loss + consonant loss + vowel loss + grapheme loss
Now to the key aspect of my solution. I realised all models perform poorly on consonant diacritic 3 (I abbreviate with cons3 in the following) and especially confuse it with cons4. And with poorly I mean game changing poorly: 89% recall compared to 99% for the other consonant classes. At first I just fought the symptoms (instead of the root cause) as several other teams did, e.g. by transferring low confident predicts from high frequent class cons4 to low frequent class cons3 and hence leveraging the definition of macro recall metric and that gave a good LB boost (0.9900 -> 0.9915). That might have been enough for gold but I wanted to improve my chances. So I wanted to understand why the model did so poor on this one and read through discussions, wikipedia and other domain knowledge related things. Luckily, I found 2 things:\n- Wikipedia: As the last member of a conjunct, য jô appears as a wavy vertical line (called যফলা jôfôla) to the right of the previous member: ক্য \"kyô\" খ্য \"khyô\" গ্য \"gyô\" ঘ্য \"ghyô\" etc. In some fonts, certain conjuncts with যফলা jôfôla appear using special fused forms: দ্য \"dyô\" ন্য \"nyô\" শ্য \"shyô\" ষ্য \"ṣyô\" স্য \"syô\" হ্য \"hyô\".
\n\nSo I realized that not only 3,4 and 6 have a jô component which might have the special jôfôla form but also that the special forms are in train for only cons4 but not for cons3, and thats why the models are so bad on cons3. My solution was the following. I recoded the cons classes into a multilabel classification problem using the following function:
\n\n```\ncons_components = ['ঁ', 'য', 'র','্']
\n\ndef is_sub(sub, lst):\n ln = len(sub)\n return any(lst[i: i + ln] == sub for i in range(len(lst) - ln + 1))
\n\ndef label2label_v2(label):\n res = np.zeros((6,),dtype=int)\n components = cons2components[label]\n for i,item in enumerate(cons_components):\n if item in components:\n res[i] = 1\n if is_sub(['্', 'য'], components):\n res[4] = 1\n if is_sub(['্', 'র'], components):\n res[5] = 1\n return res
\n\n```
\n\nThat function re-codes the 7 consonant class labels into multilabel 6 dimensional vectors
\n\n\n0 -> [0, 0, 0, 0, 0, 0]\n1 -> [1, 0, 0, 0, 0, 0]\n2 -> [0, 0, 1, 1, 0, 0]\n3 -> [0, 1, 1, 1, 1, 0]\n4 -> [0, 1, 0, 1, 1, 0]\n5 -> [0, 0, 1, 1, 0, 1]\n6 -> [0, 1, 1, 1, 1, 1]\n
That schema not only enabled to learn special forms of cons3 from cons4 but also enables to tune via thresholding (i.e. when to set a continuous prediction to 0 or 1) how the model separates the 6 classes. E.g. by setting a very low threshold on the 2nd logit more predictions would be set to 1 and hence more predictions would move from cons4 to cons3 as these two classes only differ in logit2. Purely changing the target for consonant diacritic in this way improved Public LB from 0.990 to 0.993 !
\n\nI tempered around with binarization and metric learning thresholds and purely checked LB hence I was aware of a high risk to potentially overfit to LB. So I chose my best sub and a failsafe sub that I felt was the least tempered with but still doing ok. (Both were gold on private at the end)
\n\nCheers.
", "rawMarkdown": "Thanks at Bengali.ai and kaggle for hosting this interesting competition as it was more to it than simple ensembling of computer vision models. Thanks to the authors and contributors of pytorch, pytorch-lightning, apex and pytorchcv for making my life easier.\n\n**Acknowledgements & notes on GM**\n\nI am humbled to finally become competition grandmaster. I learned a tremendous amount of tricks in the recent two years and it would not have been able if it wasn’t for all this generous sharing of top solutions and extremely talented teammates I was lucky to have along the way. \n\n\n### Short Summary\n\nMy solution is a simple bag (different seeds) of the same model seresnext50 with custom head. Major boost in LB score came from redesign of consonant diacritic target. Minor LB improvement from postprocessing tweaks such as thresholding and finding closest train examples. \n\n\n### Preprocessing\n\nI did not resize the image, the only thing I did was normalize each image by its mean and std, since I experienced a good regularization from that in previous competitions.\n\n### Architecture & Training\n\n**Backbone**\nI used a plain seresnext50 but adjusted the very first layer to replace resizing the image and account for single channel. I did that by changing input units and reducing stride from (2,2) to (1,2).\n\n```\nfrom pytorchcv.model_provider import get_model as ptcv_get_model\n\nbackbone = ptcv_get_model('seresnext50_32x4d', pretrained=True)\nbackbone = backbone.features\nbackbone.init_block.conv = nn.Conv2d(1, 64, kernel_size=(7, 7), stride=(1, 2))\n```\n\n**neck**\n\nsame as in maciejsypetkowski solution https://www.kaggle.com/c/recursion-cellular-image-classification/discussion/110543 \n\n\n**heads**\n\n- 3 heads, each for consonant, vowel and root and \n- auxiliary head for grapheme with arccos loss\n\nSo in total my used architecture looks like this:\n\n\n\n\n**Training**\n\n- train on all data\n- 4 cycle cosine annealing with augmentation increase\n - cycle 1 & 2 cutmix\n - cycle 3 & 4 cutmix, cutout, scale, translate, rotate\n- Adam \n\n**Losses**\n\n- root loss: CrossEntropy\n- consonant loss: Multi label Binary Crossentropy\n- vowel loss : CrossEntropy\n- grapheme loss: ArcCos + CrossEntropy\n\n- total loss = root loss + consonant loss + vowel loss + grapheme loss\n\n\n### The Magic (jôfôla) \nNow to the key aspect of my solution. I realised all models perform poorly on consonant diacritic 3 (I abbreviate with cons3 in the following) and especially confuse it with cons4. And with poorly I mean game changing poorly: 89% recall compared to 99% for the other consonant classes. At first I just fought the symptoms (instead of the root cause) as several other teams did, e.g. by transferring low confident predicts from high frequent class cons4 to low frequent class cons3 and hence leveraging the definition of macro recall metric and that gave a good LB boost (0.9900 -> 0.9915). That might have been enough for gold but I wanted to improve my chances. So I wanted to understand why the model did so poor on this one and read through discussions, wikipedia and other domain knowledge related things. Luckily, I found 2 things:\n- Wikipedia: As the last member of a conjunct, য jô appears as a wavy vertical line (called যফলা jôfôla) to the right of the previous member: ক্য \"kyô\" খ্য \"khyô\" গ্য \"gyô\" ঘ্য \"ghyô\" etc. In some fonts, certain conjuncts with যফলা jôfôla appear using special fused forms: দ্য \"dyô\" ন্য \"nyô\" শ্য \"shyô\" ষ্য \"ṣyô\" স্য \"syô\" হ্য \"hyô\".\n \n- The consonant diacritics are itself composed from lower level components\n - 0: []\n - 1: ['ঁ']\n - 2: ['র', '্']\n - 3: ['র', '্', 'য']\n - 4: ['্', 'য']\n - 5:['্', 'র']\n - 6: ['্', 'র', '্', 'য']\n \nSo I realized that not only 3,4 and 6 have a jô component which might have the special jôfôla form but also that the special forms are in train for **only** cons4 but **not** for cons3, and thats why the models are so bad on cons3. My solution was the following. I recoded the cons classes into a multilabel classification problem using the following function:\n\n```\ncons_components = ['ঁ', 'য', 'র','্']\n\ndef is_sub(sub, lst):\n ln = len(sub)\n return any(lst[i: i + ln] == sub for i in range(len(lst) - ln + 1))\n\ndef label2label_v2(label):\n res = np.zeros((6,),dtype=int)\n components = cons2components[label]\n for i,item in enumerate(cons_components):\n if item in components:\n res[i] = 1\n if is_sub(['্', 'য'], components):\n res[4] = 1\n if is_sub(['্', 'র'], components):\n res[5] = 1\n return res\n\n```\n\nThat function re-codes the 7 consonant class labels into multilabel 6 dimensional vectors\n\n```\n0 -> [0, 0, 0, 0, 0, 0]\n1 -> [1, 0, 0, 0, 0, 0]\n2 -> [0, 0, 1, 1, 0, 0]\n3 -> [0, 1, 1, 1, 1, 0]\n4 -> [0, 1, 0, 1, 1, 0]\n5 -> [0, 0, 1, 1, 0, 1]\n6 -> [0, 1, 1, 1, 1, 1]\n```\n\nThat schema not only enabled to learn special forms of cons3 from cons4 but also enables to tune via thresholding (i.e. when to set a continuous prediction to 0 or 1) how the model separates the 6 classes. E.g. by setting a very low threshold on the 2nd logit more predictions would be set to 1 and hence more predictions would move from cons4 to cons3 as these two classes only differ in logit2. Purely changing the target for consonant diacritic in this way improved Public LB from 0.990 to 0.993 !\n\n\n### Postprocessing\n\n- I adjusted threshold for binarizing consonant diacritic. Best was a threshold of 0.1 for all logits classes. I tried not dare to temper which each logit threshold individually as I was afraid of overfitting. Result on Public LB: 9930 -> 9937 \n- I used cosine similarity to find closest train sample for root and vowel when top1 probability < 90% and use its label. bestfitting did the same in the 1st place solution of Human Protein competition) as can be found under https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/78109 Result on Public LB: 9937 -> 9942 \n\n\n### Choosing subs\nI tempered around with binarization and metric learning thresholds and purely checked LB hence I was aware of a high risk to potentially overfit to LB. So I chose my best sub and a failsafe sub that I felt was the least tempered with but still doing ok. (Both were gold on private at the end)\n\nCheers. \n", "votes": 90 }, { "id": 780997, "postDate": "2020-03-20T19:55:32.437Z", "content": "Congratulations! So many admires to you for achieving 3*GM in such a short time! You're my role model now.😊
", "rawMarkdown": "Congratulations! So many admires to you for achieving 3*GM in such a short time! You're my role model now.😊 ", "votes": 1 }, { "id": 779331, "postDate": "2020-03-19T08:11:42.943Z", "content": "Thank you for sharing your magic, and Congrats on triple Grand Master!
", "rawMarkdown": "Thank you for sharing your magic, and Congrats on triple Grand Master!", "votes": 1 }, { "id": 778614, "postDate": "2020-03-18T15:40:12.820Z", "content": "Congratulation for the solo gold and being competition GM! You made it in only 5 weeks by solo, that's a great job!\nI learned a lot from your post and I have a question about your re-codes labels.\nIs your re-codes labels doing better than original ones for unseen graphemes? Have you tried to submit it without any post-processing?
", "rawMarkdown": "Congratulation for the solo gold and being competition GM! You made it in only 5 weeks by solo, that's a great job!\nI learned a lot from your post and I have a question about your re-codes labels.\nIs your re-codes labels doing better than original ones for unseen graphemes? Have you tried to submit it without any post-processing?", "votes": 1, "replies": [ { "id": 778741, "postDate": "2020-03-18T17:28:09.250Z", "content": "Private score (roughly) 0.92 -> 0.95, so a huge improvement
", "rawMarkdown": "Private score (roughly) 0.92 -> 0.95, so a huge improvement" }, { "id": 778748, "postDate": "2020-03-18T17:36:19.840Z", "content": "That's huge!\nI'll try it when I have my GPU free ;)
", "rawMarkdown": "That's huge!\nI'll try it when I have my GPU free ;)" } ] }, { "id": 777874, "postDate": "2020-03-18T02:04:16.210Z", "content": "Congratulations on triple GM!
", "rawMarkdown": "Congratulations on triple GM!", "votes": 1 }, { "id": 777839, "postDate": "2020-03-18T01:42:03.020Z", "content": "Congratulations on becoming the 180th Competitions GM😏 😏
", "rawMarkdown": "Congratulations on becoming the 180th Competitions GM😏 😏 ", "votes": 1 }, { "id": 777820, "postDate": "2020-03-18T01:08:15.080Z", "content": "Congrats to triple GM, consonant labeling method is interesting!
", "rawMarkdown": "Congrats to triple GM, consonant labeling method is interesting!", "votes": 1 }, { "id": 777633, "postDate": "2020-03-17T20:35:20.747Z", "content": "Congratulations @christofhenkel for the competition and for the Grand-Master status. \nCould you please elaborate and provide some reference links to help understand the following in Choosing subs:\n> I tempered around with binarization and metric learning thresholds and purely checked LB hence I was aware of a high risk to potentially overfit to LB...
\n\nThis could be really helpful for someone like me who has no idea on how to select solutions for final submission.
", "rawMarkdown": "Congratulations @christofhenkel for the competition and for the Grand-Master status. \nCould you please elaborate and provide some reference links to help understand the following in **Choosing subs**:\n> I tempered around with binarization and metric learning thresholds and purely checked LB hence I was aware of a high risk to potentially overfit to LB...\n\nThis could be really helpful for someone like me who has no idea on _how to select solutions for final submission._", "votes": 1, "replies": [ { "id": 778048, "postDate": "2020-03-18T05:29:04.777Z", "content": "Every time you use the Public Leaderboard to evaluate if something is working or not, you fit on it and that's risky, since the behavior might not hold on private. So as the second submission, I tried to chose a less fitted to leaderboard one.
", "rawMarkdown": "Every time you use the Public Leaderboard to evaluate if something is working or not, you fit on it and that's risky, since the behavior might not hold on private. So as the second submission, I tried to chose a less fitted to leaderboard one." } ] }, { "id": 776844, "postDate": "2020-03-17T17:12:12.953Z", "content": "Brilliant solution! Congrats on gold medal and grandmaster status.
\n\nRe-coding the consonants was smart. Cosine similarity is a great trick. That's a great way to use GPU multiplication and/or RAPIDS cuML kNN.
", "rawMarkdown": "Brilliant solution! Congrats on gold medal and grandmaster status.\n\nRe-coding the consonants was smart. Cosine similarity is a great trick. That's a great way to use GPU multiplication and/or RAPIDS cuML kNN.", "votes": 1 }, { "id": 776794, "postDate": "2020-03-17T16:33:08.537Z", "content": "Congrats on the result and the method. Well deserved GM title in prime!
", "rawMarkdown": "Congrats on the result and the method. Well deserved GM title in prime!\n", "votes": 1 }, { "id": 776761, "postDate": "2020-03-17T16:10:44.567Z", "content": "Wow! That's a really interesting and creative way to handle consonant 3 and 6.
", "rawMarkdown": "Wow! That's a really interesting and creative way to handle consonant 3 and 6.", "votes": 1 }, { "id": 776717, "postDate": "2020-03-17T15:33:32.863Z", "content": "Handling C=3 was key :) Congrats!
", "rawMarkdown": "Handling C=3 was key :) Congrats!", "votes": 1 }, { "id": 776712, "postDate": "2020-03-17T15:32:21.563Z", "content": "Congrats on reaching GM!
", "rawMarkdown": "Congrats on reaching GM!", "votes": 1 }, { "id": 778050, "postDate": "2020-03-18T05:31:49.187Z", "content": "Congratulations, Small question about components part, Did you set the threshold search local CV？and if we get vectors other than those seven ( for example : result =[0, 1, 1, 1, 0, 0]), what should we do?
", "rawMarkdown": "Congratulations, Small question about components part, Did you set the threshold search local CV？and if we get vectors other than those seven ( for example : result =[0, 1, 1, 1, 0, 0]), what should we do?", "votes": 2, "replies": [ { "id": 778055, "postDate": "2020-03-18T05:41:06.903Z", "content": "\n\n\nDid you set the threshold search local CV？
\n
Yes, but I also probed leaderboard a bit for that.
\n\n\n\n\nand if we get vectors other than those seven, what should we do?
\n
luckily there were not many of that cases. (Which was also a good check if the model is working properly) You could take the closest of the 7 using some distance metric (e.g. cosine distance). Here it made more sense to directly label as C=3, due to the design of the competition metric
", "rawMarkdown": "> Did you set the threshold search local CV？\n\nYes, but I also probed leaderboard a bit for that. \n\n> and if we get vectors other than those seven, what should we do?\n\nluckily there were not many of that cases. (Which was also a good check if the model is working properly) You could take the closest of the 7 using some distance metric (e.g. cosine distance). Here it made more sense to directly label as C=3, due to the design of the competition metric", "votes": 3 }, { "id": 778071, "postDate": "2020-03-18T05:57:05.337Z", "content": "Ok, Thanks for your reply
", "rawMarkdown": "Ok, Thanks for your reply" } ] }, { "id": 777986, "postDate": "2020-03-18T04:00:30.443Z", "content": "@christofhenkel Congrats to beoming triple GM, really ingenious and problem-specific solution! I remember struggling up the LB and seeing you a few places before me every step of the way during the latter quarter of the competition; chasing \"grandpa\" had been great motivation hahaha.
\n\n3 questions\n1) When you said normalizing each image by its mean and std, do you mean you apply (1) the same statistics computed on all training samples or (2) normalize every single image based on its mean and std (different for each image, kind of like InstanceNorm) ?\n2) I have never tried model with two heads (with 2 losses, like CE+arcface in this way). Do you usually get good feedbacks from this design? I have been hesitant because it's hard to find correct weighting of the losses. I wonder what is your chosen weights and how you usually choose them.\n3) Because the model is multi-output, do you leverage the multi-output for the models' prediction before post-processing? In other words, do you only use the RCV prediction or somehow combine that with grapheme prediction for inference?
\n\nThanks in advance
", "rawMarkdown": "@christofhenkel Congrats to beoming triple GM, really ingenious and problem-specific solution! I remember struggling up the LB and seeing you a few places before me every step of the way during the latter quarter of the competition; chasing \"grandpa\" had been great motivation hahaha.\n\n3 questions\n1) When you said normalizing each image by its mean and std, do you mean you apply (1) the same statistics computed on all training samples or (2) normalize every single image based on its mean and std (different for each image, kind of like InstanceNorm) ?\n2) I have never tried model with two heads (with 2 losses, like CE+arcface in this way). Do you usually get good feedbacks from this design? I have been hesitant because it's hard to find correct weighting of the losses. I wonder what is your chosen weights and how you usually choose them.\n3) Because the model is multi-output, do you leverage the multi-output for the models' prediction before post-processing? In other words, do you only use the RCV prediction or somehow combine that with grapheme prediction for inference?\n\nThanks in advance", "votes": 2, "replies": [ { "id": 778045, "postDate": "2020-03-18T05:20:14.193Z", "content": "1) normalize every single image based on its mean and std, which I thought makes sense, since you couldn't peak into the test images to norm over all samples
\n\n2) Several heads or multi losses normally work well for me. I normally start with very simple weights. and then tune them a bit. I use a simple scaling factor to get the losses to same scale, when using different losses.
\n\n3) Sometimes you can leverage from multi output. For example average the predictions, but here I did though using the grapheme prediction might hurt the models performance on unseen graphemes, so I did not try.
", "rawMarkdown": "1) normalize every single image based on its mean and std, which I thought makes sense, since you couldn't peak into the test images to norm over all samples\n\n2) Several heads or multi losses normally work well for me. I normally start with very simple weights. and then tune them a bit. I use a simple scaling factor to get the losses to same scale, when using different losses. \n\n3) Sometimes you can leverage from multi output. For example average the predictions, but here I did though using the grapheme prediction might hurt the models performance on unseen graphemes, so I did not try. ", "votes": 2 } ] }, { "id": 777698, "postDate": "2020-03-17T21:29:37.420Z", "content": "Congrats to triple GM!
", "rawMarkdown": "Congrats to triple GM!", "votes": 2 }, { "id": 776752, "postDate": "2020-03-17T15:59:22.947Z", "content": "Congrats Dieter!! You really are an inspiration to many amateurs like me. Now that you are Competition GM(and 3xGM), what would be your (realistic) advice for someone like me who wants to get better at Kaggle and become GM someday?(What is not realistic? -> don't be afraid kind of thing 😂 )
", "rawMarkdown": "Congrats Dieter!! You really are an inspiration to many amateurs like me. Now that you are Competition GM(and 3xGM), what would be your (**realistic**) advice for someone like me who wants to get better at Kaggle and become GM someday?(What is not realistic? -> don't be afraid kind of thing 😂 )", "votes": 2, "replies": [ { "id": 778062, "postDate": "2020-03-18T05:47:30.737Z", "content": "I think the young imposter in this video gives some tips. Try to learn long term and as broad as possible (NLP, computer vision, tabular data) you never know when you could use a trick from a different field.
", "rawMarkdown": "I think the young imposter in this [video](https://www.youtube.com/watch?v=Q0_Xajic_9U&feature=youtu.be ) gives some tips. Try to learn long term and as broad as possible (NLP, computer vision, tabular data) you never know when you could use a trick from a different field.", "votes": 6 } ] }, { "id": 780925, "postDate": "2020-03-20T18:36:06.823Z", "content": "Congrats, so surprised to see that you got large improvement by just using multilabel for consonant . I tried similar thing for root but not working. Also glad to see that the old gold solution from former competitions also did well in this competition.
", "rawMarkdown": "Congrats, so surprised to see that you got large improvement by just using multilabel for consonant . I tried similar thing for root but not working. Also glad to see that the old gold solution from former competitions also did well in this competition. " }, { "id": 778210, "postDate": "2020-03-18T08:28:54.523Z", "content": "Congratulations!
", "rawMarkdown": "Congratulations!" }, { "id": 782222, "postDate": "2020-03-22T03:50:58.050Z", "rawMarkdown": "", "isDeleted": true }, { "id": 781032, "postDate": "2020-03-20T20:58:48.410Z", "rawMarkdown": "", "isDeleted": true }, { "id": 780878, "postDate": "2020-03-20T17:41:06.083Z", "rawMarkdown": "", "isDeleted": true }, { "id": 777507, "postDate": "2020-03-17T18:21:34.353Z", "rawMarkdown": "", "votes": -2, "isDeleted": true } ], "comments": [ { "id": 780997, "author_name": "Helen", "author_url": "", "post_date": "2020-03-20T19:55:32.437000", "content": "Congratulations! So many admires to you for achieving 3*GM in such a short time! You're my role model now.😊
", "votes": 1, "replies": [] }, { "id": 779331, "author_name": "Tawara", "author_url": "", "post_date": "2020-03-19T08:11:42.943000", "content": "Thank you for sharing your magic, and Congrats on triple Grand Master!
", "votes": 1, "replies": [] }, { "id": 778614, "author_name": "Qishen Ha", "author_url": "", "post_date": "2020-03-18T15:40:12.820000", "content": "Congratulation for the solo gold and being competition GM! You made it in only 5 weeks by solo, that's a great job!\nI learned a lot from your post and I have a question about your re-codes labels.\nIs your re-codes labels doing better than original ones for unseen graphemes? Have you tried to submit it without any post-processing?
", "votes": 1, "replies": [ { "id": 778741, "author_name": "Dieter", "author_url": "", "post_date": "2020-03-18T17:28:09.250000", "content": "Private score (roughly) 0.92 -> 0.95, so a huge improvement
", "votes": 0, "replies": [] }, { "id": 778748, "author_name": "Qishen Ha", "author_url": "", "post_date": "2020-03-18T17:36:19.840000", "content": "That's huge!\nI'll try it when I have my GPU free ;)
", "votes": 0, "replies": [] } ] }, { "id": 777874, "author_name": "zr", "author_url": "", "post_date": "2020-03-18T02:04:16.210000", "content": "Congratulations on triple GM!
", "votes": 1, "replies": [] }, { "id": 777839, "author_name": "Gary", "author_url": "", "post_date": "2020-03-18T01:42:03.020000", "content": "Congratulations on becoming the 180th Competitions GM😏 😏
", "votes": 1, "replies": [] }, { "id": 777820, "author_name": "corochann", "author_url": "", "post_date": "2020-03-18T01:08:15.080000", "content": "Congrats to triple GM, consonant labeling method is interesting!
", "votes": 1, "replies": [] }, { "id": 777633, "author_name": "Tushar", "author_url": "", "post_date": "2020-03-17T20:35:20.747000", "content": "Congratulations @christofhenkel for the competition and for the Grand-Master status. \nCould you please elaborate and provide some reference links to help understand the following in Choosing subs:\n> I tempered around with binarization and metric learning thresholds and purely checked LB hence I was aware of a high risk to potentially overfit to LB...
\n\nThis could be really helpful for someone like me who has no idea on how to select solutions for final submission.
", "votes": 1, "replies": [ { "id": 778048, "author_name": "Dieter", "author_url": "", "post_date": "2020-03-18T05:29:04.777000", "content": "Every time you use the Public Leaderboard to evaluate if something is working or not, you fit on it and that's risky, since the behavior might not hold on private. So as the second submission, I tried to chose a less fitted to leaderboard one.
", "votes": 0, "replies": [] } ] }, { "id": 776844, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-03-17T17:12:12.953000", "content": "Brilliant solution! Congrats on gold medal and grandmaster status.
\n\nRe-coding the consonants was smart. Cosine similarity is a great trick. That's a great way to use GPU multiplication and/or RAPIDS cuML kNN.
", "votes": 1, "replies": [] }, { "id": 776794, "author_name": "CPMP", "author_url": "", "post_date": "2020-03-17T16:33:08.537000", "content": "Congrats on the result and the method. Well deserved GM title in prime!
", "votes": 1, "replies": [] }, { "id": 776761, "author_name": "Dipam Chakraborty", "author_url": "", "post_date": "2020-03-17T16:10:44.567000", "content": "Wow! That's a really interesting and creative way to handle consonant 3 and 6.
", "votes": 1, "replies": [] }, { "id": 776717, "author_name": "Psi", "author_url": "", "post_date": "2020-03-17T15:33:32.863000", "content": "Handling C=3 was key :) Congrats!
", "votes": 1, "replies": [] }, { "id": 776712, "author_name": "GreatGameDota", "author_url": "", "post_date": "2020-03-17T15:32:21.563000", "content": "Congrats on reaching GM!
", "votes": 1, "replies": [] }, { "id": 778050, "author_name": "He", "author_url": "", "post_date": "2020-03-18T05:31:49.187000", "content": "Congratulations, Small question about components part, Did you set the threshold search local CV？and if we get vectors other than those seven ( for example : result =[0, 1, 1, 1, 0, 0]), what should we do?
", "votes": 2, "replies": [ { "id": 778055, "author_name": "Dieter", "author_url": "", "post_date": "2020-03-18T05:41:06.903000", "content": "\n\n\nDid you set the threshold search local CV？
\n
Yes, but I also probed leaderboard a bit for that.
\n\n\n\n\nand if we get vectors other than those seven, what should we do?
\n
luckily there were not many of that cases. (Which was also a good check if the model is working properly) You could take the closest of the 7 using some distance metric (e.g. cosine distance). Here it made more sense to directly label as C=3, due to the design of the competition metric
", "votes": 3, "replies": [] }, { "id": 778071, "author_name": "He", "author_url": "", "post_date": "2020-03-18T05:57:05.337000", "content": "Ok, Thanks for your reply
", "votes": 0, "replies": [] } ] }, { "id": 777986, "author_name": "Nicholas Lyu", "author_url": "", "post_date": "2020-03-18T04:00:30.443000", "content": "@christofhenkel Congrats to beoming triple GM, really ingenious and problem-specific solution! I remember struggling up the LB and seeing you a few places before me every step of the way during the latter quarter of the competition; chasing \"grandpa\" had been great motivation hahaha.
\n\n3 questions\n1) When you said normalizing each image by its mean and std, do you mean you apply (1) the same statistics computed on all training samples or (2) normalize every single image based on its mean and std (different for each image, kind of like InstanceNorm) ?\n2) I have never tried model with two heads (with 2 losses, like CE+arcface in this way). Do you usually get good feedbacks from this design? I have been hesitant because it's hard to find correct weighting of the losses. I wonder what is your chosen weights and how you usually choose them.\n3) Because the model is multi-output, do you leverage the multi-output for the models' prediction before post-processing? In other words, do you only use the RCV prediction or somehow combine that with grapheme prediction for inference?
\n\nThanks in advance
", "votes": 2, "replies": [ { "id": 778045, "author_name": "Dieter", "author_url": "", "post_date": "2020-03-18T05:20:14.193000", "content": "1) normalize every single image based on its mean and std, which I thought makes sense, since you couldn't peak into the test images to norm over all samples
\n\n2) Several heads or multi losses normally work well for me. I normally start with very simple weights. and then tune them a bit. I use a simple scaling factor to get the losses to same scale, when using different losses.
\n\n3) Sometimes you can leverage from multi output. For example average the predictions, but here I did though using the grapheme prediction might hurt the models performance on unseen graphemes, so I did not try.
", "votes": 2, "replies": [] } ] }, { "id": 777698, "author_name": "Max Jeblick", "author_url": "", "post_date": "2020-03-17T21:29:37.420000", "content": "Congrats to triple GM!
", "votes": 2, "replies": [] }, { "id": 776752, "author_name": "Bibek", "author_url": "", "post_date": "2020-03-17T15:59:22.947000", "content": "Congrats Dieter!! You really are an inspiration to many amateurs like me. Now that you are Competition GM(and 3xGM), what would be your (realistic) advice for someone like me who wants to get better at Kaggle and become GM someday?(What is not realistic? -> don't be afraid kind of thing 😂 )
", "votes": 2, "replies": [ { "id": 778062, "author_name": "Dieter", "author_url": "", "post_date": "2020-03-18T05:47:30.737000", "content": "I think the young imposter in this video gives some tips. Try to learn long term and as broad as possible (NLP, computer vision, tabular data) you never know when you could use a trick from a different field.
", "votes": 6, "replies": [] } ] }, { "id": 780925, "author_name": "Morphy", "author_url": "", "post_date": "2020-03-20T18:36:06.823000", "content": "Congrats, so surprised to see that you got large improvement by just using multilabel for consonant . I tried similar thing for root but not working. Also glad to see that the old gold solution from former competitions also did well in this competition.
", "votes": 0, "replies": [] }, { "id": 778210, "author_name": "Kranti Kumar", "author_url": "", "post_date": "2020-03-18T08:28:54.523000", "content": "Congratulations!
", "votes": 0, "replies": [] }, { "id": 782222, "author_name": "", "author_url": "", "post_date": "2020-03-22T03:50:58.050000", "content": "", "votes": 0, "replies": [] }, { "id": 781032, "author_name": "", "author_url": "", "post_date": "2020-03-20T20:58:48.410000", "content": "", "votes": 0, "replies": [] }, { "id": 780878, "author_name": "", "author_url": "", "post_date": "2020-03-20T17:41:06.083000", "content": "", "votes": 0, "replies": [] }, { "id": 777507, "author_name": "", "author_url": "", "post_date": "2020-03-17T18:21:34.353000", "content": "", "votes": -2, "replies": [] } ], "raw_markdown_by_id": { "776709": "Thanks at Bengali.ai and kaggle for hosting this interesting competition as it was more to it than simple ensembling of computer vision models. Thanks to the authors and contributors of pytorch, pytorch-lightning, apex and pytorchcv for making my life easier.\n\n**Acknowledgements & notes on GM**\n\nI am humbled to finally become competition grandmaster. I learned a tremendous amount of tricks in the recent two years and it would not have been able if it wasn’t for all this generous sharing of top solutions and extremely talented teammates I was lucky to have along the way. \n\n\n### Short Summary\n\nMy solution is a simple bag (different seeds) of the same model seresnext50 with custom head. Major boost in LB score came from redesign of consonant diacritic target. Minor LB improvement from postprocessing tweaks such as thresholding and finding closest train examples. \n\n\n### Preprocessing\n\nI did not resize the image, the only thing I did was normalize each image by its mean and std, since I experienced a good regularization from that in previous competitions.\n\n### Architecture & Training\n\n**Backbone**\nI used a plain seresnext50 but adjusted the very first layer to replace resizing the image and account for single channel. I did that by changing input units and reducing stride from (2,2) to (1,2).\n\n```\nfrom pytorchcv.model_provider import get_model as ptcv_get_model\n\nbackbone = ptcv_get_model('seresnext50_32x4d', pretrained=True)\nbackbone = backbone.features\nbackbone.init_block.conv = nn.Conv2d(1, 64, kernel_size=(7, 7), stride=(1, 2))\n```\n\n**neck**\n\nsame as in maciejsypetkowski solution https://www.kaggle.com/c/recursion-cellular-image-classification/discussion/110543 \n\n\n**heads**\n\n- 3 heads, each for consonant, vowel and root and \n- auxiliary head for grapheme with arccos loss\n\nSo in total my used architecture looks like this:\n\n\n\n\n**Training**\n\n- train on all data\n- 4 cycle cosine annealing with augmentation increase\n - cycle 1 & 2 cutmix\n - cycle 3 & 4 cutmix, cutout, scale, translate, rotate\n- Adam \n\n**Losses**\n\n- root loss: CrossEntropy\n- consonant loss: Multi label Binary Crossentropy\n- vowel loss : CrossEntropy\n- grapheme loss: ArcCos + CrossEntropy\n\n- total loss = root loss + consonant loss + vowel loss + grapheme loss\n\n\n### The Magic (jôfôla) \nNow to the key aspect of my solution. I realised all models perform poorly on consonant diacritic 3 (I abbreviate with cons3 in the following) and especially confuse it with cons4. And with poorly I mean game changing poorly: 89% recall compared to 99% for the other consonant classes. At first I just fought the symptoms (instead of the root cause) as several other teams did, e.g. by transferring low confident predicts from high frequent class cons4 to low frequent class cons3 and hence leveraging the definition of macro recall metric and that gave a good LB boost (0.9900 -> 0.9915). That might have been enough for gold but I wanted to improve my chances. So I wanted to understand why the model did so poor on this one and read through discussions, wikipedia and other domain knowledge related things. Luckily, I found 2 things:\n- Wikipedia: As the last member of a conjunct, য jô appears as a wavy vertical line (called যফলা jôfôla) to the right of the previous member: ক্য \"kyô\" খ্য \"khyô\" গ্য \"gyô\" ঘ্য \"ghyô\" etc. In some fonts, certain conjuncts with যফলা jôfôla appear using special fused forms: দ্য \"dyô\" ন্য \"nyô\" শ্য \"shyô\" ষ্য \"ṣyô\" স্য \"syô\" হ্য \"hyô\".\n \n- The consonant diacritics are itself composed from lower level components\n - 0: []\n - 1: ['ঁ']\n - 2: ['র', '্']\n - 3: ['র', '্', 'য']\n - 4: ['্', 'য']\n - 5:['্', 'র']\n - 6: ['্', 'র', '্', 'য']\n \nSo I realized that not only 3,4 and 6 have a jô component which might have the special jôfôla form but also that the special forms are in train for **only** cons4 but **not** for cons3, and thats why the models are so bad on cons3. My solution was the following. I recoded the cons classes into a multilabel classification problem using the following function:\n\n```\ncons_components = ['ঁ', 'য', 'র','্']\n\ndef is_sub(sub, lst):\n ln = len(sub)\n return any(lst[i: i + ln] == sub for i in range(len(lst) - ln + 1))\n\ndef label2label_v2(label):\n res = np.zeros((6,),dtype=int)\n components = cons2components[label]\n for i,item in enumerate(cons_components):\n if item in components:\n res[i] = 1\n if is_sub(['্', 'য'], components):\n res[4] = 1\n if is_sub(['্', 'র'], components):\n res[5] = 1\n return res\n\n```\n\nThat function re-codes the 7 consonant class labels into multilabel 6 dimensional vectors\n\n```\n0 -> [0, 0, 0, 0, 0, 0]\n1 -> [1, 0, 0, 0, 0, 0]\n2 -> [0, 0, 1, 1, 0, 0]\n3 -> [0, 1, 1, 1, 1, 0]\n4 -> [0, 1, 0, 1, 1, 0]\n5 -> [0, 0, 1, 1, 0, 1]\n6 -> [0, 1, 1, 1, 1, 1]\n```\n\nThat schema not only enabled to learn special forms of cons3 from cons4 but also enables to tune via thresholding (i.e. when to set a continuous prediction to 0 or 1) how the model separates the 6 classes. E.g. by setting a very low threshold on the 2nd logit more predictions would be set to 1 and hence more predictions would move from cons4 to cons3 as these two classes only differ in logit2. Purely changing the target for consonant diacritic in this way improved Public LB from 0.990 to 0.993 !\n\n\n### Postprocessing\n\n- I adjusted threshold for binarizing consonant diacritic. Best was a threshold of 0.1 for all logits classes. I tried not dare to temper which each logit threshold individually as I was afraid of overfitting. Result on Public LB: 9930 -> 9937 \n- I used cosine similarity to find closest train sample for root and vowel when top1 probability < 90% and use its label. bestfitting did the same in the 1st place solution of Human Protein competition) as can be found under https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/78109 Result on Public LB: 9937 -> 9942 \n\n\n### Choosing subs\nI tempered around with binarization and metric learning thresholds and purely checked LB hence I was aware of a high risk to potentially overfit to LB. So I chose my best sub and a failsafe sub that I felt was the least tempered with but still doing ok. (Both were gold on private at the end)\n\nCheers. \n", "780997": "Congratulations! So many admires to you for achieving 3*GM in such a short time! You're my role model now.😊 ", "779331": "Thank you for sharing your magic, and Congrats on triple Grand Master!", "778614": "Congratulation for the solo gold and being competition GM! You made it in only 5 weeks by solo, that's a great job!\nI learned a lot from your post and I have a question about your re-codes labels.\nIs your re-codes labels doing better than original ones for unseen graphemes? Have you tried to submit it without any post-processing?", "777874": "Congratulations on triple GM!", "777839": "Congratulations on becoming the 180th Competitions GM😏 😏 ", "777820": "Congrats to triple GM, consonant labeling method is interesting!", "777633": "Congratulations @christofhenkel for the competition and for the Grand-Master status. \nCould you please elaborate and provide some reference links to help understand the following in **Choosing subs**:\n> I tempered around with binarization and metric learning thresholds and purely checked LB hence I was aware of a high risk to potentially overfit to LB...\n\nThis could be really helpful for someone like me who has no idea on _how to select solutions for final submission._", "776844": "Brilliant solution! Congrats on gold medal and grandmaster status.\n\nRe-coding the consonants was smart. Cosine similarity is a great trick. That's a great way to use GPU multiplication and/or RAPIDS cuML kNN.", "776794": "Congrats on the result and the method. Well deserved GM title in prime!\n", "776761": "Wow! That's a really interesting and creative way to handle consonant 3 and 6.", "776717": "Handling C=3 was key :) Congrats!", "776712": "Congrats on reaching GM!", "778050": "Congratulations, Small question about components part, Did you set the threshold search local CV？and if we get vectors other than those seven ( for example : result =[0, 1, 1, 1, 0, 0]), what should we do?", "777986": "@christofhenkel Congrats to beoming triple GM, really ingenious and problem-specific solution! I remember struggling up the LB and seeing you a few places before me every step of the way during the latter quarter of the competition; chasing \"grandpa\" had been great motivation hahaha.\n\n3 questions\n1) When you said normalizing each image by its mean and std, do you mean you apply (1) the same statistics computed on all training samples or (2) normalize every single image based on its mean and std (different for each image, kind of like InstanceNorm) ?\n2) I have never tried model with two heads (with 2 losses, like CE+arcface in this way). Do you usually get good feedbacks from this design? I have been hesitant because it's hard to find correct weighting of the losses. I wonder what is your chosen weights and how you usually choose them.\n3) Because the model is multi-output, do you leverage the multi-output for the models' prediction before post-processing? In other words, do you only use the RCV prediction or somehow combine that with grapheme prediction for inference?\n\nThanks in advance", "777698": "Congrats to triple GM!", "776752": "Congrats Dieter!! You really are an inspiration to many amateurs like me. Now that you are Competition GM(and 3xGM), what would be your (**realistic**) advice for someone like me who wants to get better at Kaggle and become GM someday?(What is not realistic? -> don't be afraid kind of thing 😂 )", "780925": "Congrats, so surprised to see that you got large improvement by just using multilabel for consonant . I tried similar thing for root but not working. Also glad to see that the old gold solution from former competitions also did well in this competition. ", "778210": "Congratulations!", "782222": "", "781032": "", "780878": "", "777507": "" } }