{ "id": 131734, "title": "Other 7 things that did not worked (part 2)", "url": "/competitions/bengaliai-cv19/discussion/131734", "author_name": "Vlad Vaduva", "post_date": "2020-02-21T11:38:18.799000", "votes": 15, "comment_count": 29, "views": 0, "content": "

Along the things I have shared in this topic https://www.kaggle.com/c/bengaliai-cv19/discussion/130311 I will share more of some methods, that for me did not result in a positive outcome.

\n\n
    \n
  1. Switch from Relu to LeakyRelu for avoiding dying gradient problem (no improvement and training time increased)
    2. \n
  2. Upscale image to big resolution (over 200x200)
    3. \n
  3. Too many dropout layers (sometimes enough is enough)
    4. \n
  4. Small batch sizes (my best results were in batch_size>48)
    5. \n
  5. AugMix did not improve my results (I spend some time tune the params and try to combine it with Cutmix/Mixup or Cutout)
    6. \n
  6. OHEM loss (no improvement)
    7. \n
  7. AdamW optimizer (similar results as Adam, next on the list is the old SGD)
    8. \n
", "messages": [ { "id": 752764, "postDate": "2020-02-21T11:38:18.800Z", "content": "

Along the things I have shared in this topic https://www.kaggle.com/c/bengaliai-cv19/discussion/130311 I will share more of some methods, that for me did not result in a positive outcome.

\n\n
    \n
  1. Switch from Relu to LeakyRelu for avoiding dying gradient problem (no improvement and training time increased)
    2. \n
  2. Upscale image to big resolution (over 200x200)
    3. \n
  3. Too many dropout layers (sometimes enough is enough)
    4. \n
  4. Small batch sizes (my best results were in batch_size>48)
    5. \n
  5. AugMix did not improve my results (I spend some time tune the params and try to combine it with Cutmix/Mixup or Cutout)
    6. \n
  6. OHEM loss (no improvement)
    7. \n
  7. AdamW optimizer (similar results as Adam, next on the list is the old SGD)
    8. \n
", "rawMarkdown": "Along the things I have shared in this topic https://www.kaggle.com/c/bengaliai-cv19/discussion/130311 I will share more of some methods, that for me did not result in a positive outcome.\n\n1. Switch from Relu to LeakyRelu for avoiding dying gradient problem (no improvement and training time increased)\n2. Upscale image to big resolution (over 200x200)\n3. Too many dropout layers (sometimes enough is enough)\n4. Small batch sizes (my best results were in batch_size>48)\n5. AugMix did not improve my results (I spend some time tune the params and try to combine it with Cutmix/Mixup or Cutout)\n6. OHEM loss (no improvement)\n7. AdamW optimizer (similar results as Adam, next on the list is the old SGD)", "votes": 15 }, { "id": 753030, "postDate": "2020-02-21T16:25:48.157Z", "content": "

Thanks again for sharing! Very helpful cutting down experiment times. I've had similar results on points 2,3 and 5-7.\n- I'd like to add for OHEM that I get worse results overall.\n- UPDATE: RAdam converges faster than Adam and performs better (less gap between CV,LB), next will be Lookahead SGD.\n- Seresnext101 performs better than Seresnext50 but takes longer to train.\n- Having .4 mixup, .4 cutmix and .2 aug only performs worse than just .5 mixup,cutmix.\n- UPDATE: Cutmix alone seems to perform better (less gap between CV,LB) will continue to experiment...

\n\n

My current score is using ensemble of models so I still haven't figured out the single model tricks, best single model score is .9683.

", "rawMarkdown": "Thanks again for sharing! Very helpful cutting down experiment times. I've had similar results on points 2,3 and 5-7.\n- I'd like to add for OHEM that I get worse results overall.\n- UPDATE: RAdam converges faster than Adam and performs better (less gap between CV,LB), next will be Lookahead SGD.\n- Seresnext101 performs better than Seresnext50 but takes longer to train.\n- Having .4 mixup, .4 cutmix and .2 aug only performs worse than just .5 mixup,cutmix.\n- UPDATE: Cutmix alone seems to perform better (less gap between CV,LB) will continue to experiment...\n\nMy current score is using ensemble of models so I still haven't figured out the single model tricks, best single model score is .9683.", "votes": 2, "replies": [ { "id": 753054, "postDate": "2020-02-21T17:01:43.903Z", "content": "

Good feedback @greatgamedota .\nPlease let me know how it goes with Lookahead SGD, I am really curious

\n\n

Good luck !!!

", "rawMarkdown": "Good feedback @greatgamedota .\nPlease let me know how it goes with Lookahead SGD, I am really curious\n\nGood luck !!!" }, { "id": 753065, "postDate": "2020-02-21T17:15:36.870Z", "content": "

I'm also testing OneCycle again for 300 epochs, it'll take a long time but I'll update with results

", "rawMarkdown": "I'm also testing OneCycle again for 300 epochs, it'll take a long time but I'll update with results" }, { "id": 753071, "postDate": "2020-02-21T17:19:49.290Z", "content": "

300 epochs... what hardware are you using for training ?

", "rawMarkdown": "300 epochs... what hardware are you using for training ?" }, { "id": 753073, "postDate": "2020-02-21T17:21:03.680Z", "content": "

RTX 2080 for long experiments, google colab for everything else.

", "rawMarkdown": "RTX 2080 for long experiments, google colab for everything else." }, { "id": 753749, "postDate": "2020-02-22T16:11:22.440Z", "content": "

About .4 mixup, .4 cutmix and .2 aug, could I roughly know what kind of data augmentation are you using?\nDid you use cutout on that .2 data augmentation?\nBecause .4 mixup, .4 cutmix and .2 aug is what I am doing now, I cannot get better results...

", "rawMarkdown": "About .4 mixup, .4 cutmix and .2 aug, could I roughly know what kind of data augmentation are you using?\nDid you use cutout on that .2 data augmentation?\nBecause .4 mixup, .4 cutmix and .2 aug is what I am doing now, I cannot get better results..." }, { "id": 753764, "postDate": "2020-02-22T16:25:31.003Z", "content": "

I'm using the same aug as this kernal: https://www.kaggle.com/corochann/bengali-seresnext-training-with-pytorch (the transform part not the cropping)

", "rawMarkdown": "I'm using the same aug as this kernal: https://www.kaggle.com/corochann/bengali-seresnext-training-with-pytorch (the transform part not the cropping)" }, { "id": 753802, "postDate": "2020-02-22T16:54:28.960Z", "content": "

Got it. Thank you for your sharing!

", "rawMarkdown": "Got it. Thank you for your sharing!" }, { "id": 753823, "postDate": "2020-02-22T17:19:08.847Z", "content": "

Lookahead(SGD) gives me a 0.3-0.4% accuracy boost, however I did not spend too much time testing adam because I just have more experience tweaking parameters with SGD and always seem to get better results with SGD

", "rawMarkdown": "Lookahead(SGD) gives me a 0.3-0.4% accuracy boost, however I did not spend too much time testing adam because I just have more experience tweaking parameters with SGD and always seem to get better results with SGD" }, { "id": 753837, "postDate": "2020-02-22T17:52:05.217Z", "content": "

How long did you have to train for? SGD usually converged slower than Adam when I tested it.

", "rawMarkdown": "How long did you have to train for? SGD usually converged slower than Adam when I tested it." }, { "id": 753844, "postDate": "2020-02-22T17:58:24.183Z", "content": "

I'm doing 100 epochs and using the ideas of: https://arxiv.org/pdf/1708.07120.pdf\nSo high learning rate, large batchsize and OneCycleLR

\n\n

Lookahead also helps to converge in a fewer epochs

", "rawMarkdown": "I'm doing 100 epochs and using the ideas of: https://arxiv.org/pdf/1708.07120.pdf\nSo high learning rate, large batchsize and OneCycleLR\n\nLookahead also helps to converge in a fewer epochs", "votes": 1 } ] }, { "id": 752996, "postDate": "2020-02-21T15:54:32.717Z", "content": "

tried arcface loss against 1295 grapheme class. Seemed to help vowel and consonant classification but grapheme root still suffer. Have not totally given up on arcface loss yet.

", "rawMarkdown": "tried arcface loss against 1295 grapheme class. Seemed to help vowel and consonant classification but grapheme root still suffer. Have not totally given up on arcface loss yet.\n", "votes": 1, "replies": [ { "id": 753001, "postDate": "2020-02-21T15:57:22.770Z", "content": "

Interesting, I have never use arcface loss. Please keep up to date with the progress

", "rawMarkdown": "Interesting, I have never use arcface loss. Please keep up to date with the progress" }, { "id": 753076, "postDate": "2020-02-21T17:23:18.127Z", "content": "

What kind of architecture you are using for getting features ?

", "rawMarkdown": "What kind of architecture you are using for getting features ? " }, { "id": 753195, "postDate": "2020-02-21T20:43:42.287Z", "content": "

one head for root, vowel, and consonant, and a seperate head from the feature layer trained with arcface

", "rawMarkdown": "one head for root, vowel, and consonant, and a seperate head from the feature layer trained with arcface" }, { "id": 753196, "postDate": "2020-02-21T20:43:53.903Z", "content": "

well it's not working...

", "rawMarkdown": "well it's not working..." }, { "id": 753579, "postDate": "2020-02-22T12:14:12.950Z", "content": "

unfortunately i still having figure out how to apply arcface loss properly ...

", "rawMarkdown": "unfortunately i still having figure out how to apply arcface loss properly ..." }, { "id": 753584, "postDate": "2020-02-22T12:17:17.687Z", "content": "

it sounds like you @Nirjhar Roy\n may have some valuable tip(s) for us on this topic ? please do share as I am scratching my head...😉

\n\n

I tried 2 things:\n1) use arcloss to traing feature extractor and then traing feature extractor with classification head\n2) use the model has two heads, one for classification (root, vowel, and consonant), the other for arcface loss.

\n\n

But none seems to work......

", "rawMarkdown": "it sounds like you @Nirjhar Roy\n may have some valuable tip(s) for us on this topic ? please do share as I am scratching my head...😉 \n\nI tried 2 things:\n1) use arcloss to traing feature extractor and then traing feature extractor with classification head\n2) use the model has two heads, one for classification (root, vowel, and consonant), the other for arcface loss.\n\nBut none seems to work......" } ] }, { "id": 756243, "postDate": "2020-02-25T14:45:36.077Z", "content": "

Sharing everything that does not work is also science! Thanks for sharing Vlad Vaduva

", "rawMarkdown": "Sharing everything that does not work is also science! Thanks for sharing Vlad Vaduva" }, { "id": 756222, "postDate": "2020-02-25T14:26:00.833Z", "content": "

OHEM FOCAL LOSS no improvement

", "rawMarkdown": "OHEM FOCAL LOSS no improvement" }, { "id": 752993, "postDate": "2020-02-21T15:52:16.397Z", "content": "

Thanks for sharing those!

\n\n

As I'm still in the 2nd half of the leaderboard, I don't really know if this might be any useful, but I can definitely relate to your point about Dropouts. Removing some gave me a nice boost and allowed me to get passed the 0.96 mark!

", "rawMarkdown": "Thanks for sharing those!\n\nAs I'm still in the 2nd half of the leaderboard, I don't really know if this might be any useful, but I can definitely relate to your point about Dropouts. Removing some gave me a nice boost and allowed me to get passed the 0.96 mark!" }, { "id": 752965, "postDate": "2020-02-21T15:27:19.793Z", "content": "

How you deal with grapheme_root: wrong labeling, imbalance ... etc. How's your validation score of it? I can't get more than .975 score on the only grapheme_root while others two achieve above .995+! I wonder how some people reach on it above .99!

", "rawMarkdown": "How you deal with grapheme_root: wrong labeling, imbalance ... etc. How's your validation score of it? I can't get more than .975 score on the only grapheme_root while others two achieve above .995+! I wonder how some people reach on it above .99!", "replies": [ { "id": 752979, "postDate": "2020-02-21T15:37:45.283Z", "content": "

If you are training with different folds be sure that the folds are stratified. My validation score is very similar with the leaderboard score, you seem to have a big gap between validation score and leaderboard score, my advice is: \n- try to generalize better (use dropout is you are not)\n- use more augmentation (or smarter)\n- use a smaller arhitecture of the model

", "rawMarkdown": "If you are training with different folds be sure that the folds are stratified. My validation score is very similar with the leaderboard score, you seem to have a big gap between validation score and leaderboard score, my advice is: \n- try to generalize better (use dropout is you are not)\n- use more augmentation (or smarter)\n- use a smaller arhitecture of the model", "votes": 1 }, { "id": 752988, "postDate": "2020-02-21T15:46:30.633Z", "content": "

Thanks. I am not doing different folds, just plain splitting 8:2. The validation score on grapheme_root approx 0.975 while others two are approx 0.994. I have used drop-out but as you suggest I will look carefully now. Augmentation is a big issue here. I just want to ensure if we really need to give attention to grapheme_root specifically.

", "rawMarkdown": "Thanks. I am not doing different folds, just plain splitting 8:2. The validation score on grapheme_root approx 0.975 while others two are approx 0.994. I have used drop-out but as you suggest I will look carefully now. Augmentation is a big issue here. I just want to ensure if we really need to give attention to grapheme_root specifically. " }, { "id": 752999, "postDate": "2020-02-21T15:56:15.037Z", "content": "

Try in the loss function to multiply by 2 the error from grapheme !

", "rawMarkdown": "Try in the loss function to multiply by 2 the error from grapheme ! ", "votes": 2 } ] }, { "id": 752767, "postDate": "2020-02-21T11:41:37.770Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 752776, "postDate": "2020-02-21T11:44:37.390Z", "content": "

Yes, I did but got better results with Cutmix/Mixup with cross entropy on a rexnext101 arhitecture

", "rawMarkdown": "Yes, I did but got better results with Cutmix/Mixup with cross entropy on a rexnext101 arhitecture" }, { "id": 752851, "postDate": "2020-02-21T13:24:28.337Z", "content": "

That function returns a list of [loss1, loss2, loss3], so you need to add them first and call backward() then.

", "rawMarkdown": "That function returns a list of [loss1, loss2, loss3], so you need to add them first and call backward() then.", "votes": 1 }, { "id": 752894, "postDate": "2020-02-21T14:10:53.220Z", "rawMarkdown": "", "isDeleted": true } ] } ], "comments": [ { "id": 753030, "author_name": "GreatGameDota", "author_url": "", "post_date": "2020-02-21T16:25:48.157000", "content": "

Thanks again for sharing! Very helpful cutting down experiment times. I've had similar results on points 2,3 and 5-7.\n- I'd like to add for OHEM that I get worse results overall.\n- UPDATE: RAdam converges faster than Adam and performs better (less gap between CV,LB), next will be Lookahead SGD.\n- Seresnext101 performs better than Seresnext50 but takes longer to train.\n- Having .4 mixup, .4 cutmix and .2 aug only performs worse than just .5 mixup,cutmix.\n- UPDATE: Cutmix alone seems to perform better (less gap between CV,LB) will continue to experiment...

\n\n

My current score is using ensemble of models so I still haven't figured out the single model tricks, best single model score is .9683.

", "votes": 2, "replies": [ { "id": 753054, "author_name": "Vlad Vaduva", "author_url": "", "post_date": "2020-02-21T17:01:43.903000", "content": "

Good feedback @greatgamedota .\nPlease let me know how it goes with Lookahead SGD, I am really curious

\n\n

Good luck !!!

", "votes": 0, "replies": [] }, { "id": 753065, "author_name": "GreatGameDota", "author_url": "", "post_date": "2020-02-21T17:15:36.870000", "content": "

I'm also testing OneCycle again for 300 epochs, it'll take a long time but I'll update with results

", "votes": 0, "replies": [] }, { "id": 753071, "author_name": "Vlad Vaduva", "author_url": "", "post_date": "2020-02-21T17:19:49.290000", "content": "

300 epochs... what hardware are you using for training ?

", "votes": 0, "replies": [] }, { "id": 753073, "author_name": "GreatGameDota", "author_url": "", "post_date": "2020-02-21T17:21:03.680000", "content": "

RTX 2080 for long experiments, google colab for everything else.

", "votes": 0, "replies": [] }, { "id": 753749, "author_name": "YS", "author_url": "", "post_date": "2020-02-22T16:11:22.440000", "content": "

About .4 mixup, .4 cutmix and .2 aug, could I roughly know what kind of data augmentation are you using?\nDid you use cutout on that .2 data augmentation?\nBecause .4 mixup, .4 cutmix and .2 aug is what I am doing now, I cannot get better results...

", "votes": 0, "replies": [] }, { "id": 753764, "author_name": "GreatGameDota", "author_url": "", "post_date": "2020-02-22T16:25:31.003000", "content": "

I'm using the same aug as this kernal: https://www.kaggle.com/corochann/bengali-seresnext-training-with-pytorch (the transform part not the cropping)

", "votes": 0, "replies": [] }, { "id": 753802, "author_name": "YS", "author_url": "", "post_date": "2020-02-22T16:54:28.960000", "content": "

Got it. Thank you for your sharing!

", "votes": 0, "replies": [] }, { "id": 753823, "author_name": "Yann Majewski", "author_url": "", "post_date": "2020-02-22T17:19:08.847000", "content": "

Lookahead(SGD) gives me a 0.3-0.4% accuracy boost, however I did not spend too much time testing adam because I just have more experience tweaking parameters with SGD and always seem to get better results with SGD

", "votes": 0, "replies": [] }, { "id": 753837, "author_name": "GreatGameDota", "author_url": "", "post_date": "2020-02-22T17:52:05.217000", "content": "

How long did you have to train for? SGD usually converged slower than Adam when I tested it.

", "votes": 0, "replies": [] }, { "id": 753844, "author_name": "Yann Majewski", "author_url": "", "post_date": "2020-02-22T17:58:24.183000", "content": "

I'm doing 100 epochs and using the ideas of: https://arxiv.org/pdf/1708.07120.pdf\nSo high learning rate, large batchsize and OneCycleLR

\n\n

Lookahead also helps to converge in a fewer epochs

", "votes": 1, "replies": [] } ] }, { "id": 752996, "author_name": "YL", "author_url": "", "post_date": "2020-02-21T15:54:32.717000", "content": "

tried arcface loss against 1295 grapheme class. Seemed to help vowel and consonant classification but grapheme root still suffer. Have not totally given up on arcface loss yet.

", "votes": 1, "replies": [ { "id": 753001, "author_name": "Vlad Vaduva", "author_url": "", "post_date": "2020-02-21T15:57:22.770000", "content": "

Interesting, I have never use arcface loss. Please keep up to date with the progress

", "votes": 0, "replies": [] }, { "id": 753076, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2020-02-21T17:23:18.127000", "content": "

What kind of architecture you are using for getting features ?

", "votes": 0, "replies": [] }, { "id": 753195, "author_name": "YL", "author_url": "", "post_date": "2020-02-21T20:43:42.287000", "content": "

one head for root, vowel, and consonant, and a seperate head from the feature layer trained with arcface

", "votes": 0, "replies": [] }, { "id": 753196, "author_name": "YL", "author_url": "", "post_date": "2020-02-21T20:43:53.903000", "content": "

well it's not working...

", "votes": 0, "replies": [] }, { "id": 753579, "author_name": "YL", "author_url": "", "post_date": "2020-02-22T12:14:12.950000", "content": "

unfortunately i still having figure out how to apply arcface loss properly ...

", "votes": 0, "replies": [] }, { "id": 753584, "author_name": "YL", "author_url": "", "post_date": "2020-02-22T12:17:17.687000", "content": "

it sounds like you @Nirjhar Roy\n may have some valuable tip(s) for us on this topic ? please do share as I am scratching my head...😉

\n\n

I tried 2 things:\n1) use arcloss to traing feature extractor and then traing feature extractor with classification head\n2) use the model has two heads, one for classification (root, vowel, and consonant), the other for arcface loss.

\n\n

But none seems to work......

", "votes": 0, "replies": [] } ] }, { "id": 756243, "author_name": "G-Dant", "author_url": "", "post_date": "2020-02-25T14:45:36.077000", "content": "

Sharing everything that does not work is also science! Thanks for sharing Vlad Vaduva

", "votes": 0, "replies": [] }, { "id": 756222, "author_name": "cswwp", "author_url": "", "post_date": "2020-02-25T14:26:00.833000", "content": "

OHEM FOCAL LOSS no improvement

", "votes": 0, "replies": [] }, { "id": 752993, "author_name": "Maxime Lenormand", "author_url": "", "post_date": "2020-02-21T15:52:16.397000", "content": "

Thanks for sharing those!

\n\n

As I'm still in the 2nd half of the leaderboard, I don't really know if this might be any useful, but I can definitely relate to your point about Dropouts. Removing some gave me a nice boost and allowed me to get passed the 0.96 mark!

", "votes": 0, "replies": [] }, { "id": 752965, "author_name": "Innat", "author_url": "", "post_date": "2020-02-21T15:27:19.793000", "content": "

How you deal with grapheme_root: wrong labeling, imbalance ... etc. How's your validation score of it? I can't get more than .975 score on the only grapheme_root while others two achieve above .995+! I wonder how some people reach on it above .99!

", "votes": 0, "replies": [ { "id": 752979, "author_name": "Vlad Vaduva", "author_url": "", "post_date": "2020-02-21T15:37:45.283000", "content": "

If you are training with different folds be sure that the folds are stratified. My validation score is very similar with the leaderboard score, you seem to have a big gap between validation score and leaderboard score, my advice is: \n- try to generalize better (use dropout is you are not)\n- use more augmentation (or smarter)\n- use a smaller arhitecture of the model

", "votes": 1, "replies": [] }, { "id": 752988, "author_name": "Innat", "author_url": "", "post_date": "2020-02-21T15:46:30.633000", "content": "

Thanks. I am not doing different folds, just plain splitting 8:2. The validation score on grapheme_root approx 0.975 while others two are approx 0.994. I have used drop-out but as you suggest I will look carefully now. Augmentation is a big issue here. I just want to ensure if we really need to give attention to grapheme_root specifically.

", "votes": 0, "replies": [] }, { "id": 752999, "author_name": "Vlad Vaduva", "author_url": "", "post_date": "2020-02-21T15:56:15.037000", "content": "

Try in the loss function to multiply by 2 the error from grapheme !

", "votes": 2, "replies": [] } ] }, { "id": 752767, "author_name": "", "author_url": "", "post_date": "2020-02-21T11:41:37.770000", "content": "", "votes": 0, "replies": [ { "id": 752776, "author_name": "Vlad Vaduva", "author_url": "", "post_date": "2020-02-21T11:44:37.390000", "content": "

Yes, I did but got better results with Cutmix/Mixup with cross entropy on a rexnext101 arhitecture

", "votes": 0, "replies": [] }, { "id": 752851, "author_name": "hawkey", "author_url": "", "post_date": "2020-02-21T13:24:28.337000", "content": "

That function returns a list of [loss1, loss2, loss3], so you need to add them first and call backward() then.

", "votes": 1, "replies": [] }, { "id": 752894, "author_name": "", "author_url": "", "post_date": "2020-02-21T14:10:53.220000", "content": "", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "752764": "Along the things I have shared in this topic https://www.kaggle.com/c/bengaliai-cv19/discussion/130311 I will share more of some methods, that for me did not result in a positive outcome.\n\n1. Switch from Relu to LeakyRelu for avoiding dying gradient problem (no improvement and training time increased)\n2. Upscale image to big resolution (over 200x200)\n3. Too many dropout layers (sometimes enough is enough)\n4. Small batch sizes (my best results were in batch_size>48)\n5. AugMix did not improve my results (I spend some time tune the params and try to combine it with Cutmix/Mixup or Cutout)\n6. OHEM loss (no improvement)\n7. AdamW optimizer (similar results as Adam, next on the list is the old SGD)", "753030": "Thanks again for sharing! Very helpful cutting down experiment times. I've had similar results on points 2,3 and 5-7.\n- I'd like to add for OHEM that I get worse results overall.\n- UPDATE: RAdam converges faster than Adam and performs better (less gap between CV,LB), next will be Lookahead SGD.\n- Seresnext101 performs better than Seresnext50 but takes longer to train.\n- Having .4 mixup, .4 cutmix and .2 aug only performs worse than just .5 mixup,cutmix.\n- UPDATE: Cutmix alone seems to perform better (less gap between CV,LB) will continue to experiment...\n\nMy current score is using ensemble of models so I still haven't figured out the single model tricks, best single model score is .9683.", "752996": "tried arcface loss against 1295 grapheme class. Seemed to help vowel and consonant classification but grapheme root still suffer. Have not totally given up on arcface loss yet.\n", "756243": "Sharing everything that does not work is also science! Thanks for sharing Vlad Vaduva", "756222": "OHEM FOCAL LOSS no improvement", "752993": "Thanks for sharing those!\n\nAs I'm still in the 2nd half of the leaderboard, I don't really know if this might be any useful, but I can definitely relate to your point about Dropouts. Removing some gave me a nice boost and allowed me to get passed the 0.96 mark!", "752965": "How you deal with grapheme_root: wrong labeling, imbalance ... etc. How's your validation score of it? I can't get more than .975 score on the only grapheme_root while others two achieve above .995+! I wonder how some people reach on it above .99!", "752767": "" } }