Looking through our models and submissions we found interesting thing. Two identical models, submitted about a month ago with drastically different private score. Models in question are se-resnext50, trained with same fold and absolutely identical set of augmentations, using same training schedule. Only difference is that one used weighted CE loss. Weights, calculated based on each class occurrence in train dataset were added to each of 3 loss terms.\nnp.expand_dims(1000*(1. / train.grapheme_root.value_counts()).sort_index().values, 0)\nBoth models scored very similar on local cv and public LB, 0.9911/0.0.9854 for model with weights, and 0.9909/0.9837. But weighted model scored 0.9434(hypothetical 25th place) on private, compared to only 0.9330 for model w/o weights. \nHaven't experimented further though, dropped this idea, because of very little(if any) improvement on other folds/models was observed.
Looking through our models and submissions we found interesting thing. Two identical models, submitted about a month ago with drastically different private score. Models in question are se-resnext50, trained with same fold and absolutely identical set of augmentations, using same training schedule. Only difference is that one used weighted CE loss. Weights, calculated based on each class occurrence in train dataset were added to each of 3 loss terms.\nnp.expand_dims(1000*(1. / train.grapheme_root.value_counts()).sort_index().values, 0)\nBoth models scored very similar on local cv and public LB, 0.9911/0.0.9854 for model with weights, and 0.9909/0.9837. But weighted model scored 0.9434(hypothetical 25th place) on private, compared to only 0.9330 for model w/o weights. \nHaven't experimented further though, dropped this idea, because of very little(if any) improvement on other folds/models was observed.
That is very interesting. I describe what is happening here. It appears that rare classes in public test were even more rare in private test. Therefore any encouragement to the minority classes drastically boosts private macro recall.
", "rawMarkdown": "That is very interesting. I describe what is happening [here][1]. It appears that rare classes in public test were even more rare in private test. Therefore any encouragement to the minority classes drastically boosts private macro recall.\n\n[1]: https://www.kaggle.com/c/bengaliai-cv19/discussion/136021", "votes": null }, { "id": "779298", "postDate": "03/19/2020 07:18:55", "content": "Hi CatEek, that's an interesting finding. May I also ask for more detail settings (e.g. image processing, augmentation, training schedule, training labels) of your se-resnext50 models? I was struggling to train a se-resnext 50 that can achieve > 0.98 LB and want to understand what's wrong. Thanks a lot!
", "rawMarkdown": "Hi CatEek, that's an interesting finding. May I also ask for more detail settings (e.g. image processing, augmentation, training schedule, training labels) of your se-resnext50 models? I was struggling to train a se-resnext 50 that can achieve > 0.98 LB and want to understand what's wrong. Thanks a lot!", "votes": null }, { "id": "779355", "postDate": "03/19/2020 08:54:48", "content": "Hi, Just a plain se-resnext50, single head with 186 units. It used original resolution, cutmix with 0.3 probability and one of 4 cutout variations, 1-2 small holes/10-20 large holes, black/white color with probability of 0.2. Adam with reduce on plateau with initial LR 4e-4
", "rawMarkdown": "Hi, Just a plain se-resnext50, single head with 186 units. It used original resolution, cutmix with 0.3 probability and one of 4 cutout variations, 1-2 small holes/10-20 large holes, black/white color with probability of 0.2. Adam with reduce on plateau with initial LR 4e-4", "votes": null }, { "id": "779372", "postDate": "03/19/2020 09:21:27", "content": "Your code is incorrect, with imbalanced parenthesis. Something may be missing.
\n\nWe used this in some of our models, and it helps indeed.
\n\nFYI, there has been some discussions in the forum about class weights, and how to cap them to avoid too large weights.
", "rawMarkdown": "Your code is incorrect, with imbalanced parenthesis. Something may be missing.\n\nWe used this in some of our models, and it helps indeed.\n\nFYI, there has been some discussions in the forum about class weights, and how to cap them to avoid too large weights.", "votes": null }, { "id": "779457", "postDate": "03/19/2020 11:22:43", "content": "Thanks for your attention, indeed I forgot a small part of code. Fixed now
", "rawMarkdown": "Thanks for your attention, indeed I forgot a small part of code. Fixed now", "votes": null } ], "comments": [ { "id": 779008, "author_name": "cdeotte", "author_url": "", "post_date": "03/18/2020 23:37:33", "content": "That is very interesting. I describe what is happening here. It appears that rare classes in public test were even more rare in private test. Therefore any encouragement to the minority classes drastically boosts private macro recall.
", "votes": null, "replies": [] }, { "id": 779298, "author_name": "hugotong6425", "author_url": "", "post_date": "03/19/2020 07:18:55", "content": "Hi CatEek, that's an interesting finding. May I also ask for more detail settings (e.g. image processing, augmentation, training schedule, training labels) of your se-resnext50 models? I was struggling to train a se-resnext 50 that can achieve > 0.98 LB and want to understand what's wrong. Thanks a lot!
", "votes": null, "replies": [ { "id": 779355, "author_name": "cateek", "author_url": "", "post_date": "03/19/2020 08:54:48", "content": "Hi, Just a plain se-resnext50, single head with 186 units. It used original resolution, cutmix with 0.3 probability and one of 4 cutout variations, 1-2 small holes/10-20 large holes, black/white color with probability of 0.2. Adam with reduce on plateau with initial LR 4e-4
", "votes": null, "replies": [] } ] }, { "id": 779372, "author_name": "cpmpml", "author_url": "", "post_date": "03/19/2020 09:21:27", "content": "Your code is incorrect, with imbalanced parenthesis. Something may be missing.
\n\nWe used this in some of our models, and it helps indeed.
\n\nFYI, there has been some discussions in the forum about class weights, and how to cap them to avoid too large weights.
", "votes": null, "replies": [ { "id": 779457, "author_name": "cateek", "author_url": "", "post_date": "03/19/2020 11:22:43", "content": "Thanks for your attention, indeed I forgot a small part of code. Fixed now
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "778878": "Looking through our models and submissions we found interesting thing. Two identical models, submitted about a month ago with drastically different private score. Models in question are se-resnext50, trained with same fold and absolutely identical set of augmentations, using same training schedule. Only difference is that one used weighted CE loss. Weights, calculated based on each class occurrence in train dataset were added to each of 3 loss terms.\n`np.expand_dims(1000*(1. / train.grapheme_root.value_counts()).sort_index().values, 0)`\nBoth models scored very similar on local cv and public LB, 0.9911/0.0.9854 for model with weights, and 0.9909/0.9837. But weighted model scored 0.9434(hypothetical 25th place) on private, compared to only 0.9330 for model w/o weights. \nHaven't experimented further though, dropped this idea, because of very little(if any) improvement on other folds/models was observed.", "779008": "That is very interesting. I describe what is happening [here][1]. It appears that rare classes in public test were even more rare in private test. Therefore any encouragement to the minority classes drastically boosts private macro recall.\n\n[1]: https://www.kaggle.com/c/bengaliai-cv19/discussion/136021", "779298": "Hi CatEek, that's an interesting finding. May I also ask for more detail settings (e.g. image processing, augmentation, training schedule, training labels) of your se-resnext50 models? I was struggling to train a se-resnext 50 that can achieve > 0.98 LB and want to understand what's wrong. Thanks a lot!", "779355": "Hi, Just a plain se-resnext50, single head with 186 units. It used original resolution, cutmix with 0.3 probability and one of 4 cutout variations, 1-2 small holes/10-20 large holes, black/white color with probability of 0.2. Adam with reduce on plateau with initial LR 4e-4", "779372": "Your code is incorrect, with imbalanced parenthesis. Something may be missing.\n\nWe used this in some of our models, and it helps indeed.\n\nFYI, there has been some discussions in the forum about class weights, and how to cap them to avoid too large weights.", "779457": "Thanks for your attention, indeed I forgot a small part of code. Fixed now" }, "source": "meta" }