I started taking part in this competition some time ago and no matter what I do, I can't exceed 0.96 on the leaderboard and 0.965 on local validation.
\n\nI have tried the following things:\n* various backbones from resnet50 to se_resnext_50;\n* various ways to add pooling: changing pooling of the backbone or adding pooling after the backbone;\n* starting from different learning rate: 0.1-0.0001;\n* using no augmentations or using a lot of augmentations;\n* creating images with iofass approach or simply pasting reshaped arrays on a square (make_square function), using different sizes from 128x128 up to 256x256;\n* various optimizers and schedulers;\n* using mixup callback from catalyst;\n* training up to 150 epochs;\n* training the whole net from the start or training only the head at the beginning and then the whole net;
\n\nAnd none of it helped.
\n\nMaybe I'm missing some basic thing?
", "messages": [ { "id": 750471, "postDate": "2020-02-19T12:20:48.203Z", "content": "I started taking part in this competition some time ago and no matter what I do, I can't exceed 0.96 on the leaderboard and 0.965 on local validation.
\n\nI have tried the following things:\n* various backbones from resnet50 to se_resnext_50;\n* various ways to add pooling: changing pooling of the backbone or adding pooling after the backbone;\n* starting from different learning rate: 0.1-0.0001;\n* using no augmentations or using a lot of augmentations;\n* creating images with iofass approach or simply pasting reshaped arrays on a square (make_square function), using different sizes from 128x128 up to 256x256;\n* various optimizers and schedulers;\n* using mixup callback from catalyst;\n* training up to 150 epochs;\n* training the whole net from the start or training only the head at the beginning and then the whole net;
\n\nAnd none of it helped.
\n\nMaybe I'm missing some basic thing?
", "rawMarkdown": "I started taking part in this competition some time ago and no matter what I do, I can't exceed 0.96 on the leaderboard and 0.965 on local validation.\n\nI have tried the following things:\n* various backbones from resnet50 to se_resnext_50;\n* various ways to add pooling: changing pooling of the backbone or adding pooling after the backbone;\n* starting from different learning rate: 0.1-0.0001;\n* using no augmentations or using a lot of augmentations;\n* creating images with iofass approach or simply pasting reshaped arrays on a square (make_square function), using different sizes from 128x128 up to 256x256;\n* various optimizers and schedulers;\n* using mixup callback from catalyst;\n* training up to 150 epochs;\n* training the whole net from the start or training only the head at the beginning and then the whole net;\n\nAnd none of it helped.\n\nMaybe I'm missing some basic thing?", "votes": 44 }, { "id": 751880, "postDate": "2020-02-20T15:46:21.387Z", "content": "My thanks to everyone who answered me :)\nIt seems that the following points were important:
\n\nWhat is your best single model lb now? just curious on your results haha :)
", "rawMarkdown": "What is your best single model lb now? just curious on your results haha :)", "votes": 1 }, { "id": 751895, "postDate": "2020-02-20T16:04:56.363Z", "content": "what does 'heavier head of the net' mean?
", "rawMarkdown": "what does 'heavier head of the net' mean?", "votes": 1 }, { "id": 751904, "postDate": "2020-02-20T16:16:13.810Z", "content": "fancier combo of things for linear/classification layers?
", "rawMarkdown": "fancier combo of things for linear/classification layers?", "votes": 1 }, { "id": 752072, "postDate": "2020-02-20T18:28:23.457Z", "content": "Maybe we will share our score soon :)
\n\nYes, \"heavier head of the net\" means more and different layers on the top of the backbone.
", "rawMarkdown": "Maybe we will share our score soon :)\n\n\nYes, \"heavier head of the net\" means more and different layers on the top of the backbone.", "votes": 3 } ] }, { "id": 750679, "postDate": "2020-02-19T15:50:24.953Z", "content": "This competition so far has to me felt very similar to MNIST, where so many people are getting high scores s.t. I'm not sure if we are simply rolling dice on the test randomness or it's genuine improvement. :/
\n\nAs a suggestion, have you tried ensembling? Though it's usually a last minute thing I find the very long training time makes it somewhat timely to do it earlier just in case.
", "rawMarkdown": "This competition so far has to me felt very similar to MNIST, where so many people are getting high scores s.t. I'm not sure if we are simply rolling dice on the test randomness or it's genuine improvement. :/ \n\nAs a suggestion, have you tried ensembling? Though it's usually a last minute thing I find the very long training time makes it somewhat timely to do it earlier just in case. ", "votes": 7, "replies": [ { "id": 750878, "postDate": "2020-02-19T19:15:32.237Z", "content": "Best advice for sure! :)\nThanks
", "rawMarkdown": "Best advice for sure! :)\nThanks", "votes": 1 } ] }, { "id": 750479, "postDate": "2020-02-19T12:29:26.847Z", "content": "i think you have wrong augmentations, \nwith out any augmentations, it is easy to get cv 0.96 - 0.97.
\n\nand your init lr is too large.
", "rawMarkdown": "i think you have wrong augmentations, \nwith out any augmentations, it is easy to get cv 0.96 - 0.97.\n\nand your init lr is too large.", "votes": 7, "replies": [ { "id": 752545, "postDate": "2020-02-21T06:12:55.130Z", "content": "Is 0.0001 too large ?
", "rawMarkdown": "Is 0.0001 too large ? " } ] }, { "id": 750507, "postDate": "2020-02-19T13:00:09.777Z", "content": "cutmix+resnet34+autoaugment can achieve cv 0.976 with 80 epoch
", "rawMarkdown": "cutmix+resnet34+autoaugment can achieve cv 0.976 with 80 epoch ", "votes": 7, "replies": [ { "id": 751057, "postDate": "2020-02-20T00:56:29.040Z", "content": "@masterzj I wonder what batch size is this using? 80 epochs is very short!~
", "rawMarkdown": "@masterzj I wonder what batch size is this using? 80 epochs is very short!~", "votes": 1 }, { "id": 751131, "postDate": "2020-02-20T02:35:19.760Z", "content": "128 is enough.
", "rawMarkdown": "128 is enough.", "votes": 3 }, { "id": 751135, "postDate": "2020-02-20T02:39:23.937Z", "content": "@masterzj what are the autoaugment features you used?
", "rawMarkdown": "@masterzj what are the autoaugment features you used?", "votes": 1 }, { "id": 751142, "postDate": "2020-02-20T02:52:58.253Z", "content": "@mobassir just ImageNetPolicy
", "rawMarkdown": "@mobassir just ImageNetPolicy", "votes": 2 }, { "id": 751360, "postDate": "2020-02-20T06:47:31.617Z", "content": "I actually got res34 CV 0.975, but the LB was 0.965..\nse-resnet had smaller CV-LB differences (like (0.02).
", "rawMarkdown": "I actually got res34 CV 0.975, but the LB was 0.965..\nse-resnet had smaller CV-LB differences (like (0.02).", "votes": 3 }, { "id": 751373, "postDate": "2020-02-20T06:58:14.643Z", "content": "@kyoshioka47 what settings did you use? Optimizer , LR , img size , and augmentations? 0.965 with resnet34 is a really good score. How many epochs did you train for?
", "rawMarkdown": "@kyoshioka47 what settings did you use? Optimizer , LR , img size , and augmentations? 0.965 with resnet34 is a really good score. How many epochs did you train for?", "votes": 1 } ] }, { "id": 750527, "postDate": "2020-02-19T13:27:17.520Z", "content": "For me, cutmix/mixup helped a lot on a rexnext101 arhitecture, now I am combining cutmix with some affine transformers\nThere are some interesting public kernel with decent results (over 0.96), you can check them out\nGood luck
", "rawMarkdown": "For me, cutmix/mixup helped a lot on a rexnext101 arhitecture, now I am combining cutmix with some affine transformers\nThere are some interesting public kernel with decent results (over 0.96), you can check them out\nGood luck", "votes": 5, "replies": [ { "id": 750578, "postDate": "2020-02-19T14:23:54.173Z", "content": "we need deep architecture to make cutmix/mixup works fine or resnext50 works well too?
", "rawMarkdown": "we need deep architecture to make cutmix/mixup works fine or resnext50 works well too? ", "votes": 2 }, { "id": 750590, "postDate": "2020-02-19T14:34:15.910Z", "content": "resnext50/resnext101 worked perfectly for me, especially 101
", "rawMarkdown": "resnext50/resnext101 worked perfectly for me, especially 101", "votes": 4 }, { "id": 751063, "postDate": "2020-02-20T01:03:13.427Z", "content": "I'm using a xresnet34 and cutmix is working well with 100 epochs! So idk what is the minimum architecture size
", "rawMarkdown": "I'm using a xresnet34 and cutmix is working well with 100 epochs! So idk what is the minimum architecture size", "votes": 1 }, { "id": 751145, "postDate": "2020-02-20T03:00:32.967Z", "content": "Can i know whats your image size? @vladvdv
", "rawMarkdown": "Can i know whats your image size? @vladvdv ", "votes": 2 }, { "id": 751382, "postDate": "2020-02-20T07:09:54.957Z", "content": "@yannmajewski What's xresnet34 and how's it different from resnet34?
", "rawMarkdown": "@yannmajewski What's xresnet34 and how's it different from resnet34?", "votes": 1 }, { "id": 751523, "postDate": "2020-02-20T08:54:29.677Z", "content": "I suppose that xresnet is fastai version:\nhttps://github.com/fastai/fastai/blob/master/fastai/vision/models/xresnet.py
", "rawMarkdown": "I suppose that xresnet is fastai version:\nhttps://github.com/fastai/fastai/blob/master/fastai/vision/models/xresnet.py", "votes": 1 }, { "id": 751857, "postDate": "2020-02-20T15:21:53.963Z", "content": "@cdeotte Yes just like @artgor said, its the fast ai implementation of a resnet34 with some tweaks from https://arxiv.org/abs/1812.01187 if i remember correctly
", "rawMarkdown": "@cdeotte Yes just like @artgor said, its the fast ai implementation of a resnet34 with some tweaks from https://arxiv.org/abs/1812.01187 if i remember correctly", "votes": 2 }, { "id": 752812, "postDate": "2020-02-21T12:34:15.713Z", "content": "@vladvdv Hey! Did you use the implementation of cutmix shared in this discussion?https://www.kaggle.com/c/bengaliai-cv19/discussion/126504
", "rawMarkdown": "@vladvdv Hey! Did you use the implementation of cutmix shared in this discussion?https://www.kaggle.com/c/bengaliai-cv19/discussion/126504" } ] }, { "id": 752748, "postDate": "2020-02-21T11:14:56.073Z", "content": "My CV score is 0.978 without cutmix and mixup, but when I add cutmix or mixup on my model, cv score is hard to break 0.975...
", "rawMarkdown": "My CV score is 0.978 without cutmix and mixup, but when I add cutmix or mixup on my model, cv score is hard to break 0.975...", "votes": 3, "replies": [ { "id": 752757, "postDate": "2020-02-21T11:26:16.697Z", "content": "Try to improve the publicly available cutmix implementation. For me, the alpha (0.4) did not work. And I changed it, so it generates different cuts for every sample in the batch.
", "rawMarkdown": "Try to improve the publicly available cutmix implementation. For me, the alpha (0.4) did not work. And I changed it, so it generates different cuts for every sample in the batch. ", "votes": 8 }, { "id": 752760, "postDate": "2020-02-21T11:31:34.073Z", "content": "The cutmix's paper shows alpha=1 is best, May I ask generates different cuts helps your score? when I use cutmix or mixup, the training loss became very unstable
", "rawMarkdown": "The cutmix's paper shows alpha=1 is best, May I ask generates different cuts helps your score? when I use cutmix or mixup, the training loss became very unstable", "votes": 2 }, { "id": 752773, "postDate": "2020-02-21T11:43:46.973Z", "content": "I only run one experiment with the public implementation, since then I use different cuts, It helped, but I am not 100% that was the reason. The default alpha is not worked for me at all. I tried with 0.4 (public) and 1.0 (official), but neither is the best, in my case.\nTry to output some images, you will see the issues. There are too small cuts, background only cuts, etc..
", "rawMarkdown": "I only run one experiment with the public implementation, since then I use different cuts, It helped, but I am not 100% that was the reason. The default alpha is not worked for me at all. I tried with 0.4 (public) and 1.0 (official), but neither is the best, in my case.\nTry to output some images, you will see the issues. There are too small cuts, background only cuts, etc..", "votes": 2 }, { "id": 752778, "postDate": "2020-02-21T11:45:36.760Z", "content": "ok, Thanks for your sharing
", "rawMarkdown": "ok, Thanks for your sharing", "votes": 2 }, { "id": 753020, "postDate": "2020-02-21T16:08:53.573Z", "content": "Also alpha=0.4 better than alpha=1.0
", "rawMarkdown": "Also alpha=0.4 better than alpha=1.0", "votes": 1 } ] }, { "id": 752641, "postDate": "2020-02-21T08:45:34.113Z", "content": "😂 I've tried a lot of things to improve the score but never succeeded to cross 0.97 score for a single model. It's really confusing to know so many people can easily get such a high score.
", "rawMarkdown": "😂 I've tried a lot of things to improve the score but never succeeded to cross 0.97 score for a single model. It's really confusing to know so many people can easily get such a high score.", "votes": 3 }, { "id": 752292, "postDate": "2020-02-20T20:53:40.567Z", "content": "@artgor On lighter note I can only suggest to readers: \"Here is the list of techniques that you may apply, should you get stuck somewhere. Besides, please don't forget to pick and choose from comments as well\". 😄
", "rawMarkdown": "@artgor On lighter note I can only suggest to readers: \"Here is the list of techniques that you may apply, should you get stuck somewhere. Besides, please don't forget to pick and choose from comments as well\". 😄 ", "votes": 3, "replies": [ { "id": 752504, "postDate": "2020-02-21T04:54:12.870Z", "content": "Yes, agree :)
", "rawMarkdown": "Yes, agree :)", "votes": 1 } ] }, { "id": 751699, "postDate": "2020-02-20T12:20:58.070Z", "content": "@artgor How have you come out from the stuck? We have seen that you have achieved a reasonable score now! What was the issue?
", "rawMarkdown": "@artgor How have you come out from the stuck? We have seen that you have achieved a reasonable score now! What was the issue?", "votes": 3 }, { "id": 751643, "postDate": "2020-02-20T11:13:29.840Z", "content": "Hi @artgor . Even we are in the same situation as urs. But I see that u have jumped to decent lb. Can you share what u were doing wrong and how u improved ur score so much ?
", "rawMarkdown": "Hi @artgor . Even we are in the same situation as urs. But I see that u have jumped to decent lb. Can you share what u were doing wrong and how u improved ur score so much ?", "votes": 3, "replies": [ { "id": 751753, "postDate": "2020-02-20T13:29:55.227Z", "content": "Perhaps, merging with a good team helped a bit :)
", "rawMarkdown": "Perhaps, merging with a good team helped a bit :)", "votes": 4 } ] }, { "id": 751338, "postDate": "2020-02-20T06:11:11.723Z", "content": "Some classes of grapheme_root like 84 and 61 are very similar. I think the key point is to make model distinguish them correctly. Still need other tricks to make it happen. I don't think one can achieve by just using augment/cutmix/mixup etc.
", "rawMarkdown": "Some classes of grapheme_root like 84 and 61 are very similar. I think the key point is to make model distinguish them correctly. Still need other tricks to make it happen. I don't think one can achieve by just using augment/cutmix/mixup etc. ", "votes": 3 }, { "id": 751014, "postDate": "2020-02-19T23:36:58.760Z", "content": "Finally got past the bubble though it was using an ensemble of 3 models so I'm still kinda stuck. How do people score so high with only one model
", "rawMarkdown": "Finally got past the bubble though it was using an ensemble of 3 models so I'm still kinda stuck. How do people score so high with only one model", "votes": 3, "replies": [ { "id": 751043, "postDate": "2020-02-20T00:37:51.527Z", "content": "What was the LB score of your individual model before ensembling?
", "rawMarkdown": "What was the LB score of your individual model before ensembling?", "votes": 1 }, { "id": 751055, "postDate": "2020-02-20T00:50:18.177Z", "content": "```\nCV: .9779\nLB: .9675
\n\nCV: .9773\nLB: .968
\n\nCV: .97691\nLB: .9683
\n\nEmsemble: .9696\n```
", "rawMarkdown": "```\nCV: .9779\nLB: .9675\n\nCV: .9773\nLB: .968\n\nCV: .97691\nLB: .9683\n\nEmsemble: .9696\n```", "votes": 5 }, { "id": 751062, "postDate": "2020-02-20T01:02:07.453Z", "content": "Thanks!
", "rawMarkdown": "Thanks!", "votes": 1 }, { "id": 751377, "postDate": "2020-02-20T07:05:30.443Z", "content": "Great job GreatGameDota
", "rawMarkdown": "Great job GreatGameDota", "votes": 1 }, { "id": 752107, "postDate": "2020-02-20T18:44:53.487Z", "content": "Removing the worst model increased score even more: .9699
", "rawMarkdown": "Removing the worst model increased score even more: .9699" }, { "id": 754347, "postDate": "2020-02-23T12:51:31.943Z", "content": "These results are quite similar to mine.
", "rawMarkdown": "These results are quite similar to mine." } ] }, { "id": 751009, "postDate": "2020-02-19T23:16:58.763Z", "content": "128x128 -> 160x160 definitely helped a little bit for me, seresnext50 -> seresnext101 also helped tiny little bit, but appeared to be overfitting a little, despite cutmix and mixup and affine.
\n\nhaven't tried really long training session, nor different optimizer. but from what I read the optimizer choice maybe less important. sheduler and learning rate seems to play an role in this. From my observation, there are a lot of local optimum, i.e. the same training loss see wide range of validation loss, but I have not figure out anything.
", "rawMarkdown": "128x128 -> 160x160 definitely helped a little bit for me, seresnext50 -> seresnext101 also helped tiny little bit, but appeared to be overfitting a little, despite cutmix and mixup and affine.\n\nhaven't tried really long training session, nor different optimizer. but from what I read the optimizer choice maybe less important. sheduler and learning rate seems to play an role in this. From my observation, there are a lot of local optimum, i.e. the same training loss see wide range of validation loss, but I have not figure out anything.", "votes": 3 }, { "id": 750636, "postDate": "2020-02-19T14:59:36.543Z", "content": "same thing! I am pretty stuck
", "rawMarkdown": "same thing! I am pretty stuck", "votes": 3, "replies": [ { "id": 750999, "postDate": "2020-02-19T22:50:03.867Z", "content": "well you guys are definitely less stuck than me... gpu none stop for 3-4 weeks, still havent got anywhere...
", "rawMarkdown": "well you guys are definitely less stuck than me... gpu none stop for 3-4 weeks, still havent got anywhere...", "votes": 1 }, { "id": 754606, "postDate": "2020-02-23T20:31:55.183Z", "content": "How did you get unstuck? :p
", "rawMarkdown": "How did you get unstuck? :p", "votes": 1 } ] }, { "id": 753192, "postDate": "2020-02-21T20:37:34.443Z", "content": "Good work!
", "rawMarkdown": "Good work!", "votes": 1, "replies": [ { "id": 755205, "postDate": "2020-02-24T14:50:28.877Z", "content": "yes indeed, very nice
", "rawMarkdown": "yes indeed, very nice" } ] }, { "id": 753167, "postDate": "2020-02-21T19:42:18.593Z", "content": "Same problem
", "rawMarkdown": "Same problem\n\n", "votes": 1 }, { "id": 751704, "postDate": "2020-02-20T12:28:33.403Z", "content": "Hi @artgor well done!
", "rawMarkdown": "Hi @artgor well done!", "votes": 1 }, { "id": 751601, "postDate": "2020-02-20T10:22:31.387Z", "content": "I only use a resnet34 with Radam,mixup,4-folds,some transforms and mish activation.\nwhich get 0.962 on CV and 0.965 on LB.\nHavn't try any bigger model.But I am wondering why people using a bigger model,since the resnet34 is not bad....
", "rawMarkdown": "I only use a resnet34 with Radam,mixup,4-folds,some transforms and mish activation.\nwhich get 0.962 on CV and 0.965 on LB.\nHavn't try any bigger model.But I am wondering why people using a bigger model,since the resnet34 is not bad....", "votes": 1 }, { "id": 751305, "postDate": "2020-02-20T05:03:17.640Z", "content": "Same. We have tried everything we could , and most of the things mentioned in discussions which gave people a boost in scores, but can't get past 0.97 CV :(
\n\n", "rawMarkdown": "Same. We have tried everything we could , and most of the things mentioned in discussions which gave people a boost in scores, but can't get past 0.97 CV :(\n\n@virajbagal ", "votes": 1 }, { "id": 750915, "postDate": "2020-02-19T20:23:34.427Z", "content": "@artgor Did you find the issue causing your problem? I have noticed you pretty get a reasonable score on LB.
", "rawMarkdown": "@artgor Did you find the issue causing your problem? I have noticed you pretty get a reasonable score on LB. ", "votes": 1 }, { "id": 751173, "postDate": "2020-02-20T03:32:45.423Z", "content": "After playing deep neural networks for almost one year especially on image processing, I feel like I've learned a million knowledge...out of infinity💪
", "rawMarkdown": "After playing deep neural networks for almost one year especially on image processing, I feel like I've learned a million knowledge...out of infinity💪 ", "votes": 2 }, { "id": 750600, "postDate": "2020-02-19T14:37:54.580Z", "content": "Same here. Actually It seems pretty hard to obtain minimum ~97 lb score. There is only 19 days left! People are suggesting the open-secret stuff I would say, to get easily high score on lb, but I don't think so. With almost same some guys are easily getting top scores and others are not. \nFew things I have to re-check: \n- am I doing regularization properly? lr, shed! \n- is my used augmentation method good enough for such data sets?
", "rawMarkdown": "Same here. Actually It seems pretty hard to obtain minimum ~97 lb score. There is only 19 days left! People are suggesting the open-secret stuff I would say, to get easily high score on lb, but I don't think so. With almost same some guys are easily getting top scores and others are not. \nFew things I have to re-check: \n- am I doing regularization properly? lr, shed! \n- is my used augmentation method good enough for such data sets?", "votes": 2 }, { "id": 750581, "postDate": "2020-02-19T14:26:16.480Z", "content": "I just achieved 0.96 with only resnet34 and 25 epochs on 128x128 imgs.\nMb bad seed? )
", "rawMarkdown": "I just achieved 0.96 with only resnet34 and 25 epochs on 128x128 imgs.\nMb bad seed? )", "votes": 2 }, { "id": 752295, "postDate": "2020-02-20T20:59:02.437Z", "content": "Interesting. I have 0.989 CV but only 0.969～ LB.
", "rawMarkdown": "Interesting. I have 0.989 CV but only 0.969～ LB.", "votes": 1, "replies": [ { "id": 756109, "postDate": "2020-02-25T12:22:42.723Z", "content": "Did you check for leaks? My CVs and LBs are consistent.
", "rawMarkdown": "Did you check for leaks? My CVs and LBs are consistent." } ] }, { "id": 754187, "postDate": "2020-02-23T07:27:11.680Z", "content": "how you all are having Discussion expert and masters tag while I am still on the contributor, lol.
", "rawMarkdown": "how you all are having Discussion expert and masters tag while I am still on the contributor, lol. ", "votes": -1 }, { "id": 754243, "postDate": "2020-02-23T09:17:18.817Z", "content": "me too
", "rawMarkdown": "me too" }, { "id": 754176, "postDate": "2020-02-23T07:06:00.693Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 751880, "author_name": "Andrey Lukyanenko", "author_url": "", "post_date": "2020-02-20T15:46:21.387000", "content": "My thanks to everyone who answered me :)\nIt seems that the following points were important:
\n\nWhat is your best single model lb now? just curious on your results haha :)
", "votes": 1, "replies": [] }, { "id": 751895, "author_name": "Viraj Bagal", "author_url": "", "post_date": "2020-02-20T16:04:56.363000", "content": "what does 'heavier head of the net' mean?
", "votes": 1, "replies": [] }, { "id": 751904, "author_name": "YL", "author_url": "", "post_date": "2020-02-20T16:16:13.810000", "content": "fancier combo of things for linear/classification layers?
", "votes": 1, "replies": [] }, { "id": 752072, "author_name": "Andrey Lukyanenko", "author_url": "", "post_date": "2020-02-20T18:28:23.457000", "content": "Maybe we will share our score soon :)
\n\nYes, \"heavier head of the net\" means more and different layers on the top of the backbone.
", "votes": 3, "replies": [] } ] }, { "id": 750679, "author_name": "Brian Lee", "author_url": "", "post_date": "2020-02-19T15:50:24.953000", "content": "This competition so far has to me felt very similar to MNIST, where so many people are getting high scores s.t. I'm not sure if we are simply rolling dice on the test randomness or it's genuine improvement. :/
\n\nAs a suggestion, have you tried ensembling? Though it's usually a last minute thing I find the very long training time makes it somewhat timely to do it earlier just in case.
", "votes": 7, "replies": [ { "id": 750878, "author_name": "Mobassir", "author_url": "", "post_date": "2020-02-19T19:15:32.237000", "content": "Best advice for sure! :)\nThanks
", "votes": 1, "replies": [] } ] }, { "id": 750479, "author_name": "Finlay", "author_url": "", "post_date": "2020-02-19T12:29:26.847000", "content": "i think you have wrong augmentations, \nwith out any augmentations, it is easy to get cv 0.96 - 0.97.
\n\nand your init lr is too large.
", "votes": 7, "replies": [ { "id": 752545, "author_name": "ikki1111", "author_url": "", "post_date": "2020-02-21T06:12:55.130000", "content": "Is 0.0001 too large ?
", "votes": 0, "replies": [] } ] }, { "id": 750507, "author_name": "Bet4Honor", "author_url": "", "post_date": "2020-02-19T13:00:09.777000", "content": "cutmix+resnet34+autoaugment can achieve cv 0.976 with 80 epoch
", "votes": 7, "replies": [ { "id": 751057, "author_name": "Nicholas Lyu", "author_url": "", "post_date": "2020-02-20T00:56:29.040000", "content": "@masterzj I wonder what batch size is this using? 80 epochs is very short!~
", "votes": 1, "replies": [] }, { "id": 751131, "author_name": "Bet4Honor", "author_url": "", "post_date": "2020-02-20T02:35:19.760000", "content": "128 is enough.
", "votes": 3, "replies": [] }, { "id": 751135, "author_name": "Mobassir", "author_url": "", "post_date": "2020-02-20T02:39:23.937000", "content": "@masterzj what are the autoaugment features you used?
", "votes": 1, "replies": [] }, { "id": 751142, "author_name": "Bet4Honor", "author_url": "", "post_date": "2020-02-20T02:52:58.253000", "content": "@mobassir just ImageNetPolicy
", "votes": 2, "replies": [] }, { "id": 751360, "author_name": "arutema47", "author_url": "", "post_date": "2020-02-20T06:47:31.617000", "content": "I actually got res34 CV 0.975, but the LB was 0.965..\nse-resnet had smaller CV-LB differences (like (0.02).
", "votes": 3, "replies": [] }, { "id": 751373, "author_name": "Satwik", "author_url": "", "post_date": "2020-02-20T06:58:14.643000", "content": "@kyoshioka47 what settings did you use? Optimizer , LR , img size , and augmentations? 0.965 with resnet34 is a really good score. How many epochs did you train for?
", "votes": 1, "replies": [] } ] }, { "id": 750527, "author_name": "Vlad Vaduva", "author_url": "", "post_date": "2020-02-19T13:27:17.520000", "content": "For me, cutmix/mixup helped a lot on a rexnext101 arhitecture, now I am combining cutmix with some affine transformers\nThere are some interesting public kernel with decent results (over 0.96), you can check them out\nGood luck
", "votes": 5, "replies": [ { "id": 750578, "author_name": "Kupchanski", "author_url": "", "post_date": "2020-02-19T14:23:54.173000", "content": "we need deep architecture to make cutmix/mixup works fine or resnext50 works well too?
", "votes": 2, "replies": [] }, { "id": 750590, "author_name": "Vlad Vaduva", "author_url": "", "post_date": "2020-02-19T14:34:15.910000", "content": "resnext50/resnext101 worked perfectly for me, especially 101
", "votes": 4, "replies": [] }, { "id": 751063, "author_name": "Yann Majewski", "author_url": "", "post_date": "2020-02-20T01:03:13.427000", "content": "I'm using a xresnet34 and cutmix is working well with 100 epochs! So idk what is the minimum architecture size
", "votes": 1, "replies": [] }, { "id": 751145, "author_name": "ratan rohith", "author_url": "", "post_date": "2020-02-20T03:00:32.967000", "content": "Can i know whats your image size? @vladvdv
", "votes": 2, "replies": [] }, { "id": 751382, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-02-20T07:09:54.957000", "content": "@yannmajewski What's xresnet34 and how's it different from resnet34?
", "votes": 1, "replies": [] }, { "id": 751523, "author_name": "Andrey Lukyanenko", "author_url": "", "post_date": "2020-02-20T08:54:29.677000", "content": "I suppose that xresnet is fastai version:\nhttps://github.com/fastai/fastai/blob/master/fastai/vision/models/xresnet.py
", "votes": 1, "replies": [] }, { "id": 751857, "author_name": "Yann Majewski", "author_url": "", "post_date": "2020-02-20T15:21:53.963000", "content": "@cdeotte Yes just like @artgor said, its the fast ai implementation of a resnet34 with some tweaks from https://arxiv.org/abs/1812.01187 if i remember correctly
", "votes": 2, "replies": [] }, { "id": 752812, "author_name": "Yann Majewski", "author_url": "", "post_date": "2020-02-21T12:34:15.713000", "content": "@vladvdv Hey! Did you use the implementation of cutmix shared in this discussion?https://www.kaggle.com/c/bengaliai-cv19/discussion/126504
", "votes": 0, "replies": [] } ] }, { "id": 752748, "author_name": "He", "author_url": "", "post_date": "2020-02-21T11:14:56.073000", "content": "My CV score is 0.978 without cutmix and mixup, but when I add cutmix or mixup on my model, cv score is hard to break 0.975...
", "votes": 3, "replies": [ { "id": 752757, "author_name": "Peter", "author_url": "", "post_date": "2020-02-21T11:26:16.697000", "content": "Try to improve the publicly available cutmix implementation. For me, the alpha (0.4) did not work. And I changed it, so it generates different cuts for every sample in the batch.
", "votes": 8, "replies": [] }, { "id": 752760, "author_name": "He", "author_url": "", "post_date": "2020-02-21T11:31:34.073000", "content": "The cutmix's paper shows alpha=1 is best, May I ask generates different cuts helps your score? when I use cutmix or mixup, the training loss became very unstable
", "votes": 2, "replies": [] }, { "id": 752773, "author_name": "Peter", "author_url": "", "post_date": "2020-02-21T11:43:46.973000", "content": "I only run one experiment with the public implementation, since then I use different cuts, It helped, but I am not 100% that was the reason. The default alpha is not worked for me at all. I tried with 0.4 (public) and 1.0 (official), but neither is the best, in my case.\nTry to output some images, you will see the issues. There are too small cuts, background only cuts, etc..
", "votes": 2, "replies": [] }, { "id": 752778, "author_name": "He", "author_url": "", "post_date": "2020-02-21T11:45:36.760000", "content": "ok, Thanks for your sharing
", "votes": 2, "replies": [] }, { "id": 753020, "author_name": "cswwp", "author_url": "", "post_date": "2020-02-21T16:08:53.573000", "content": "Also alpha=0.4 better than alpha=1.0
", "votes": 1, "replies": [] } ] }, { "id": 752641, "author_name": "syoya", "author_url": "", "post_date": "2020-02-21T08:45:34.113000", "content": "😂 I've tried a lot of things to improve the score but never succeeded to cross 0.97 score for a single model. It's really confusing to know so many people can easily get such a high score.
", "votes": 3, "replies": [] }, { "id": 752292, "author_name": "Solve for Fun", "author_url": "", "post_date": "2020-02-20T20:53:40.567000", "content": "@artgor On lighter note I can only suggest to readers: \"Here is the list of techniques that you may apply, should you get stuck somewhere. Besides, please don't forget to pick and choose from comments as well\". 😄
", "votes": 3, "replies": [ { "id": 752504, "author_name": "Andrey Lukyanenko", "author_url": "", "post_date": "2020-02-21T04:54:12.870000", "content": "Yes, agree :)
", "votes": 1, "replies": [] } ] }, { "id": 751699, "author_name": "Mohammad Azam Khan", "author_url": "", "post_date": "2020-02-20T12:20:58.070000", "content": "@artgor How have you come out from the stuck? We have seen that you have achieved a reasonable score now! What was the issue?
", "votes": 3, "replies": [] }, { "id": 751643, "author_name": "Viraj Bagal", "author_url": "", "post_date": "2020-02-20T11:13:29.840000", "content": "Hi @artgor . Even we are in the same situation as urs. But I see that u have jumped to decent lb. Can you share what u were doing wrong and how u improved ur score so much ?
", "votes": 3, "replies": [ { "id": 751753, "author_name": "Victor Zaguskin", "author_url": "", "post_date": "2020-02-20T13:29:55.227000", "content": "Perhaps, merging with a good team helped a bit :)
", "votes": 4, "replies": [] } ] }, { "id": 751338, "author_name": "Morphy", "author_url": "", "post_date": "2020-02-20T06:11:11.723000", "content": "Some classes of grapheme_root like 84 and 61 are very similar. I think the key point is to make model distinguish them correctly. Still need other tricks to make it happen. I don't think one can achieve by just using augment/cutmix/mixup etc.
", "votes": 3, "replies": [] }, { "id": 751014, "author_name": "GreatGameDota", "author_url": "", "post_date": "2020-02-19T23:36:58.760000", "content": "Finally got past the bubble though it was using an ensemble of 3 models so I'm still kinda stuck. How do people score so high with only one model
", "votes": 3, "replies": [ { "id": 751043, "author_name": "Cyr1ll", "author_url": "", "post_date": "2020-02-20T00:37:51.527000", "content": "What was the LB score of your individual model before ensembling?
", "votes": 1, "replies": [] }, { "id": 751055, "author_name": "GreatGameDota", "author_url": "", "post_date": "2020-02-20T00:50:18.177000", "content": "```\nCV: .9779\nLB: .9675
\n\nCV: .9773\nLB: .968
\n\nCV: .97691\nLB: .9683
\n\nEmsemble: .9696\n```
", "votes": 5, "replies": [] }, { "id": 751062, "author_name": "Cyr1ll", "author_url": "", "post_date": "2020-02-20T01:02:07.453000", "content": "Thanks!
", "votes": 1, "replies": [] }, { "id": 751377, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-02-20T07:05:30.443000", "content": "Great job GreatGameDota
", "votes": 1, "replies": [] }, { "id": 752107, "author_name": "GreatGameDota", "author_url": "", "post_date": "2020-02-20T18:44:53.487000", "content": "Removing the worst model increased score even more: .9699
", "votes": 0, "replies": [] }, { "id": 754347, "author_name": "syoya", "author_url": "", "post_date": "2020-02-23T12:51:31.943000", "content": "These results are quite similar to mine.
", "votes": 0, "replies": [] } ] }, { "id": 751009, "author_name": "YL", "author_url": "", "post_date": "2020-02-19T23:16:58.763000", "content": "128x128 -> 160x160 definitely helped a little bit for me, seresnext50 -> seresnext101 also helped tiny little bit, but appeared to be overfitting a little, despite cutmix and mixup and affine.
\n\nhaven't tried really long training session, nor different optimizer. but from what I read the optimizer choice maybe less important. sheduler and learning rate seems to play an role in this. From my observation, there are a lot of local optimum, i.e. the same training loss see wide range of validation loss, but I have not figure out anything.
", "votes": 3, "replies": [] }, { "id": 750636, "author_name": "Nicholas Lyu", "author_url": "", "post_date": "2020-02-19T14:59:36.543000", "content": "same thing! I am pretty stuck
", "votes": 3, "replies": [ { "id": 750999, "author_name": "YL", "author_url": "", "post_date": "2020-02-19T22:50:03.867000", "content": "well you guys are definitely less stuck than me... gpu none stop for 3-4 weeks, still havent got anywhere...
", "votes": 1, "replies": [] }, { "id": 754606, "author_name": "Yann Majewski", "author_url": "", "post_date": "2020-02-23T20:31:55.183000", "content": "How did you get unstuck? :p
", "votes": 1, "replies": [] } ] }, { "id": 753192, "author_name": "srgrvy", "author_url": "", "post_date": "2020-02-21T20:37:34.443000", "content": "Good work!
", "votes": 1, "replies": [ { "id": 755205, "author_name": "Chryfi", "author_url": "", "post_date": "2020-02-24T14:50:28.877000", "content": "yes indeed, very nice
", "votes": 0, "replies": [] } ] }, { "id": 753167, "author_name": "PIkachu", "author_url": "", "post_date": "2020-02-21T19:42:18.593000", "content": "Same problem
", "votes": 1, "replies": [] }, { "id": 751704, "author_name": "Mohamed Ebrahim", "author_url": "", "post_date": "2020-02-20T12:28:33.403000", "content": "Hi @artgor well done!
", "votes": 1, "replies": [] }, { "id": 751601, "author_name": "Gongchongwei", "author_url": "", "post_date": "2020-02-20T10:22:31.387000", "content": "I only use a resnet34 with Radam,mixup,4-folds,some transforms and mish activation.\nwhich get 0.962 on CV and 0.965 on LB.\nHavn't try any bigger model.But I am wondering why people using a bigger model,since the resnet34 is not bad....
", "votes": 1, "replies": [] }, { "id": 751305, "author_name": "Satwik", "author_url": "", "post_date": "2020-02-20T05:03:17.640000", "content": "Same. We have tried everything we could , and most of the things mentioned in discussions which gave people a boost in scores, but can't get past 0.97 CV :(
\n\n", "votes": 1, "replies": [] }, { "id": 750915, "author_name": "Innat", "author_url": "", "post_date": "2020-02-19T20:23:34.427000", "content": "@artgor Did you find the issue causing your problem? I have noticed you pretty get a reasonable score on LB.
", "votes": 1, "replies": [] }, { "id": 751173, "author_name": "Endi Niu", "author_url": "", "post_date": "2020-02-20T03:32:45.423000", "content": "After playing deep neural networks for almost one year especially on image processing, I feel like I've learned a million knowledge...out of infinity💪
", "votes": 2, "replies": [] }, { "id": 750600, "author_name": "Innat", "author_url": "", "post_date": "2020-02-19T14:37:54.580000", "content": "Same here. Actually It seems pretty hard to obtain minimum ~97 lb score. There is only 19 days left! People are suggesting the open-secret stuff I would say, to get easily high score on lb, but I don't think so. With almost same some guys are easily getting top scores and others are not. \nFew things I have to re-check: \n- am I doing regularization properly? lr, shed! \n- is my used augmentation method good enough for such data sets?
", "votes": 2, "replies": [] }, { "id": 750581, "author_name": "Kupchanski", "author_url": "", "post_date": "2020-02-19T14:26:16.480000", "content": "I just achieved 0.96 with only resnet34 and 25 epochs on 128x128 imgs.\nMb bad seed? )
", "votes": 2, "replies": [] }, { "id": 752295, "author_name": "Gold Retriever", "author_url": "", "post_date": "2020-02-20T20:59:02.437000", "content": "Interesting. I have 0.989 CV but only 0.969～ LB.
", "votes": 1, "replies": [ { "id": 756109, "author_name": "Quan", "author_url": "", "post_date": "2020-02-25T12:22:42.723000", "content": "Did you check for leaks? My CVs and LBs are consistent.
", "votes": 0, "replies": [] } ] }, { "id": 754187, "author_name": "shivam purbia", "author_url": "", "post_date": "2020-02-23T07:27:11.680000", "content": "how you all are having Discussion expert and masters tag while I am still on the contributor, lol.
", "votes": -1, "replies": [] }, { "id": 754243, "author_name": "Tian", "author_url": "", "post_date": "2020-02-23T09:17:18.817000", "content": "me too
", "votes": 0, "replies": [] }, { "id": 754176, "author_name": "", "author_url": "", "post_date": "2020-02-23T07:06:00.693000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "750471": "I started taking part in this competition some time ago and no matter what I do, I can't exceed 0.96 on the leaderboard and 0.965 on local validation.\n\nI have tried the following things:\n* various backbones from resnet50 to se_resnext_50;\n* various ways to add pooling: changing pooling of the backbone or adding pooling after the backbone;\n* starting from different learning rate: 0.1-0.0001;\n* using no augmentations or using a lot of augmentations;\n* creating images with iofass approach or simply pasting reshaped arrays on a square (make_square function), using different sizes from 128x128 up to 256x256;\n* various optimizers and schedulers;\n* using mixup callback from catalyst;\n* training up to 150 epochs;\n* training the whole net from the start or training only the head at the beginning and then the whole net;\n\nAnd none of it helped.\n\nMaybe I'm missing some basic thing?", "751880": "My thanks to everyone who answered me :)\nIt seems that the following points were important:\n\n* a heavier head of the net;\n* bigger batch;\n* better augmentation;\n* longer training;\n* little changes to parameters;\n* teaming up;", "750679": "This competition so far has to me felt very similar to MNIST, where so many people are getting high scores s.t. I'm not sure if we are simply rolling dice on the test randomness or it's genuine improvement. :/ \n\nAs a suggestion, have you tried ensembling? Though it's usually a last minute thing I find the very long training time makes it somewhat timely to do it earlier just in case. ", "750479": "i think you have wrong augmentations, \nwith out any augmentations, it is easy to get cv 0.96 - 0.97.\n\nand your init lr is too large.", "750507": "cutmix+resnet34+autoaugment can achieve cv 0.976 with 80 epoch ", "750527": "For me, cutmix/mixup helped a lot on a rexnext101 arhitecture, now I am combining cutmix with some affine transformers\nThere are some interesting public kernel with decent results (over 0.96), you can check them out\nGood luck", "752748": "My CV score is 0.978 without cutmix and mixup, but when I add cutmix or mixup on my model, cv score is hard to break 0.975...", "752641": "😂 I've tried a lot of things to improve the score but never succeeded to cross 0.97 score for a single model. It's really confusing to know so many people can easily get such a high score.", "752292": "@artgor On lighter note I can only suggest to readers: \"Here is the list of techniques that you may apply, should you get stuck somewhere. Besides, please don't forget to pick and choose from comments as well\". 😄 ", "751699": "@artgor How have you come out from the stuck? We have seen that you have achieved a reasonable score now! What was the issue?", "751643": "Hi @artgor . Even we are in the same situation as urs. But I see that u have jumped to decent lb. Can you share what u were doing wrong and how u improved ur score so much ?", "751338": "Some classes of grapheme_root like 84 and 61 are very similar. I think the key point is to make model distinguish them correctly. Still need other tricks to make it happen. I don't think one can achieve by just using augment/cutmix/mixup etc. ", "751014": "Finally got past the bubble though it was using an ensemble of 3 models so I'm still kinda stuck. How do people score so high with only one model", "751009": "128x128 -> 160x160 definitely helped a little bit for me, seresnext50 -> seresnext101 also helped tiny little bit, but appeared to be overfitting a little, despite cutmix and mixup and affine.\n\nhaven't tried really long training session, nor different optimizer. but from what I read the optimizer choice maybe less important. sheduler and learning rate seems to play an role in this. From my observation, there are a lot of local optimum, i.e. the same training loss see wide range of validation loss, but I have not figure out anything.", "750636": "same thing! I am pretty stuck", "753192": "Good work!", "753167": "Same problem\n\n", "751704": "Hi @artgor well done!", "751601": "I only use a resnet34 with Radam,mixup,4-folds,some transforms and mish activation.\nwhich get 0.962 on CV and 0.965 on LB.\nHavn't try any bigger model.But I am wondering why people using a bigger model,since the resnet34 is not bad....", "751305": "Same. We have tried everything we could , and most of the things mentioned in discussions which gave people a boost in scores, but can't get past 0.97 CV :(\n\n@virajbagal ", "750915": "@artgor Did you find the issue causing your problem? I have noticed you pretty get a reasonable score on LB. ", "751173": "After playing deep neural networks for almost one year especially on image processing, I feel like I've learned a million knowledge...out of infinity💪 ", "750600": "Same here. Actually It seems pretty hard to obtain minimum ~97 lb score. There is only 19 days left! People are suggesting the open-secret stuff I would say, to get easily high score on lb, but I don't think so. With almost same some guys are easily getting top scores and others are not. \nFew things I have to re-check: \n- am I doing regularization properly? lr, shed! \n- is my used augmentation method good enough for such data sets?", "750581": "I just achieved 0.96 with only resnet34 and 25 epochs on 128x128 imgs.\nMb bad seed? )", "752295": "Interesting. I have 0.989 CV but only 0.969～ LB.", "754187": "how you all are having Discussion expert and masters tag while I am still on the contributor, lol. ", "754243": "me too", "754176": "" } }