![\"\"](\"https://i.imgur.com/iB1dxkk.png\")
As I see improvements in my local CV, I also notice a strange tradeoff in the CV scores of grapheme_root, vowel_diacritic and consonant_diacritic.
Improvement in the score of grapheme_root leads to relative decrease in the scores of vowel_diacritic and consonant_diacritic, and vice-versa. So model with high score of grapheme_root but with low overall CV may score better on LB compared to the model with high CV and low grapheme_root score. This is what exactly happened to me recently, first time when improvement in CV did not correlate to LB increase.
So make sure you split your CV strategy into overall CV and class scores
", "messages": [ { "id": 720273, "postDate": "2020-01-16T09:36:55.910Z", "content": "As I see improvements in my local CV, I also notice a strange tradeoff in the CV scores of grapheme_root, vowel_diacritic and consonant_diacritic.
Improvement in the score of grapheme_root leads to relative decrease in the scores of vowel_diacritic and consonant_diacritic, and vice-versa. So model with high score of grapheme_root but with low overall CV may score better on LB compared to the model with high CV and low grapheme_root score. This is what exactly happened to me recently, first time when improvement in CV did not correlate to LB increase.
So make sure you split your CV strategy into overall CV and class scores
", "rawMarkdown": "As I see improvements in my local CV, I also notice a strange tradeoff in the CV scores of `grapheme_root`, `vowel_diacritic` and `consonant_diacritic`. \n\nImprovement in the score of `grapheme_root` leads to relative decrease in the scores of `vowel_diacritic` and `consonant_diacritic`, and vice-versa. So model with high score of `grapheme_root` but with low overall CV may score better on LB compared to the model with high CV and low `grapheme_root` score. This is what exactly happened to me recently, first time when improvement in CV did not correlate to LB increase. \n\nSo make sure you split your CV strategy into overall CV and class scores\n", "votes": 6 }, { "id": 723248, "postDate": "2020-01-19T18:19:30.753Z", "content": "@bibek777 it might be due to the learning rate, if you have the same for each head. I saw with a basic resnet18 that training with a high learning rate like 0.1 made the vowel_diacritic and consonant_diacritic learn well on first 5 epochs, while grapheme_root was doing worse than random systematically.
Not sure if this is just because of different number of classes or if it's really something.\nNot sure what can be done either : \n- different learning rates for different heads (but if you have just a linear layer in each head it probably won't help much)\n- freeze two heads and fine tune the last one alternatively on each type (vowel, consonant or grapheme - per batch or per epoch maybe) with different learning rates, this seems a bit weird but I wonder if anyone tried it.
", "rawMarkdown": "@bibek777 it might be due to the learning rate, if you have the same for each head. I saw with a basic resnet18 that training with a high learning rate like 0.1 made the `vowel_diacritic` and `consonant_diacritic` learn well on first 5 epochs, while `grapheme_root` was doing worse than random systematically.\n\nNot sure if this is just because of different number of classes or if it's really something.\nNot sure what can be done either : \n- different learning rates for different heads (but if you have just a linear layer in each head it probably won't help much)\n- freeze two heads and fine tune the last one alternatively on each type (vowel, consonant or grapheme - per batch or per epoch maybe) with different learning rates, this seems a bit weird but I wonder if anyone tried it.", "votes": 1 }, { "id": 720318, "postDate": "2020-01-16T10:17:13.213Z", "content": "@bibek777 what do you mean by splitting your CV strategy into overall CV and class scores is the way forward
", "rawMarkdown": "@bibek777 what do you mean by splitting your CV strategy into overall CV and class scores is the way forward", "votes": 1, "replies": [ { "id": 720323, "postDate": "2020-01-16T10:20:33.330Z", "content": "something like this:\n\nOverall CV: XXX\ngrapheme_root : xxx\nvowel_diacritic: yyy\nconsonant_diacritic: zzz\n
\n\n\nLooks like we have more grapheme_root samples
\n
The number of samples of all three components is going to be equal.
\nbut looks like grapheme root part is more difficult to classify in the test set as compared to the training set.
\n\n\nThe number of samples of all three components is going to be equal.
\n
what makes you think that??
", "rawMarkdown": "> The number of samples of all three components is going to be equal.\n\nwhat makes you think that??" }, { "id": 720319, "postDate": "2020-01-16T10:17:54.647Z", "content": "because each test image contains all the three components.
", "rawMarkdown": "because each test image contains all the three components.", "votes": 2 }, { "id": 720325, "postDate": "2020-01-16T10:21:34.567Z", "content": "Thanx!! I will update my post
", "rawMarkdown": "Thanx!! I will update my post" }, { "id": 720334, "postDate": "2020-01-16T10:28:01.703Z", "content": "but this doesn't explain why increase in the scores of grapheme_root leads to decrease in scores of vowel_diacritic and consonant_diacritic, and vice-versa?
@dhananjay3 I am sorry , how do we know that all test images contain all three component ? if consonant_diacritics and vowel_diacritics are 0 that means there are no such component to detect right ? Does that not make it \"Class Not Present\" for some samples ?\ne.g If we create a classifier with vowel_diacritic = 0 and non-zero , will it not become a binary classification problem? Whereas , there is always a grapeheme_root present in an image .
", "rawMarkdown": "@dhananjay3 I am sorry , how do we know that all test images contain all three component ? if consonant_diacritics and vowel_diacritics are 0 that means there are no such component to detect right ? Does that not make it \"Class Not Present\" for some samples ?\ne.g If we create a classifier with vowel_diacritic = 0 and non-zero , will it not become a binary classification problem? Whereas , there is always a grapeheme_root present in an image ." }, { "id": 723692, "postDate": "2020-01-20T10:42:36.080Z", "content": "@phoenix9032 0 does mean that image does not contain consonant/vowel but we still have to predict that it is not there in the image and score for this will be included in the recall. so it is just another class to classify into.
", "rawMarkdown": "@phoenix9032 0 does mean that image does not contain consonant/vowel but we still have to predict that it is not there in the image and score for this will be included in the recall. so it is just another class to classify into.", "votes": 2 } ] }, { "id": 720307, "postDate": "2020-01-16T10:08:37.100Z", "content": "In such case, do you think that bagging models that would respectively perform their best on either grapheme_root or vowel_diacritic and consonant_diacritic could be a good idea ? I have not tested it myself for now !
that could be a nice thing to try once you run out of ideas. for now I would recommend you to improve your single model for all classes
", "rawMarkdown": "that could be a nice thing to try once you run out of ideas. for now I would recommend you to improve your single model for all classes" } ] }, { "id": 724870, "postDate": "2020-01-21T15:30:25.220Z", "content": "@bibek777 Even I am facing the same problem, I think there are few damaged images (improper crop or improper labelling) which decreases our score !!! as mentioned here - https://www.kaggle.com/c/bengaliai-cv19/discussion/126833 by Nihar. I am personally trying different model together for vowel and consonant and a separate model for graphemes
", "rawMarkdown": "@bibek777 Even I am facing the same problem, I think there are few damaged images (improper crop or improper labelling) which decreases our score !!! as mentioned here - https://www.kaggle.com/c/bengaliai-cv19/discussion/126833 by Nihar. I am personally trying different model together for vowel and consonant and a separate model for graphemes" }, { "id": 723908, "postDate": "2020-01-20T15:41:33.653Z", "content": "I have also tried to reduce not the val_loss but the grapheme_root_val_loss which seemed to give some interesting results. I didn't necessarily measure everything correctly (which I need to start getting in the habit of doing though) but it might be worth investigating more in-depth.
I have been seeing something similar. Maybe adding some more layers which are separate would help (I'm guessing you have a single fully connected layer for each output...)?\n
why not have two separate models? \nOne for grapheme and other for the vowel and consonant? (I am pretty confident this would yield good results, but would increase submission time drastically)
", "rawMarkdown": "why not have two separate models? \nOne for grapheme and other for the vowel and consonant? (I am pretty confident this would yield good results, but would increase submission time drastically)", "replies": [ { "id": 721187, "postDate": "2020-01-17T06:38:04.770Z", "rawMarkdown": "", "isDeleted": true }, { "id": 745433, "postDate": "2020-02-13T20:07:53.163Z", "content": "\n\n\nMulti-task learning (MTL) is a subfield of machine learning in which multiple learning tasks are solved at the same time, while exploiting commonalities and differences across tasks. This can result in improved learning efficiency and prediction accuracy for the task-specific models, when compared to training the models separately. Early versions of MTL were called \"hints\".
\n
https://en.wikipedia.org/wiki/Multi-task_learning
\n\nPlus, as you've noted, increase compute, increased train time, increased inference time, increase ram resources, etc.
", "rawMarkdown": "> Multi-task learning (MTL) is a subfield of machine learning in which multiple learning tasks are solved at the same time, while exploiting commonalities and differences across tasks. This can result in improved learning efficiency and prediction accuracy for the task-specific models, when compared to training the models separately. Early versions of MTL were called \"hints\". \n\nhttps://en.wikipedia.org/wiki/Multi-task_learning\n\nPlus, as you've noted, increase compute, increased train time, increased inference time, increase ram resources, etc." } ] }, { "id": 720721, "postDate": "2020-01-16T17:12:21.990Z", "rawMarkdown": "", "votes": -1, "isDeleted": true, "replies": [ { "id": 720764, "postDate": "2020-01-16T17:56:10.593Z", "content": "\n\n\nyou should be able to prove class distributions of popular combinations to see if they are consistently different from the train set
\n
things are getting interesting now!!!
", "rawMarkdown": "> you should be able to prove class distributions of popular combinations to see if they are consistently different from the train set\n\nthings are getting interesting now!!!" }, { "id": 720782, "postDate": "2020-01-16T18:13:11.513Z", "content": "Looks like failed submissions are correctly applied towards daily limits, but aren't properly showing up in the displayed submission count. I'll file a bug report, but unlimited probing should be infeasible.
", "rawMarkdown": "Looks like failed submissions are correctly applied towards daily limits, but aren't properly showing up in the displayed submission count. I'll file a bug report, but unlimited probing should be infeasible.", "votes": 3 }, { "id": 720786, "postDate": "2020-01-16T18:17:44.040Z", "rawMarkdown": "", "votes": -1, "isDeleted": true }, { "id": 720791, "postDate": "2020-01-16T18:28:34.577Z", "content": "I don't think private probing is possible in this competition. I mean, technically, what you wrote is possible, but without a good model, you don't have the information you'd like to leak. (Or am I missing something?)
", "rawMarkdown": "I don't think private probing is possible in this competition. I mean, technically, what you wrote is possible, but without a good model, you don't have the information you'd like to leak. (Or am I missing something?)" }, { "id": 720795, "postDate": "2020-01-16T18:39:04.067Z", "rawMarkdown": "", "isDeleted": true }, { "id": 721083, "postDate": "2020-01-17T03:04:06.610Z", "content": "I didn't even thought about probing public LB test set.\nThe most important thing is that you might not be able to tell whether private LB test set has a same distribution as public LB test set.\nI think probing public LB can only help you overfitting to public LB and nothing more.
", "rawMarkdown": "I didn't even thought about probing public LB test set.\nThe most important thing is that you might not be able to tell whether private LB test set has a same distribution as public LB test set.\nI think probing public LB can only help you overfitting to public LB and nothing more.", "votes": 6 }, { "id": 721135, "postDate": "2020-01-17T04:57:31.557Z", "content": "probing test dataset is a competition trick. i think it is not required here. i estimated that by using ML methods, 0.99+ is possible for private test set as i suggest using normal ML methods is sufficient.
\n\non a side note, you can take a look at the kaggle google doodle competition on how knowing the number of class instances can improve results (i.e. greedy selection of the top results for k classes)
", "rawMarkdown": "probing test dataset is a competition trick. i think it is not required here. i estimated that by using ML methods, 0.99+ is possible for private test set as i suggest using normal ML methods is sufficient.\n\non a side note, you can take a look at the kaggle google doodle competition on how knowing the number of class instances can improve results (i.e. greedy selection of the top results for k classes)" }, { "id": 722607, "postDate": "2020-01-18T19:53:55.790Z", "content": "This might be an unpopular opinion, but I would hope that we could stay away from such practices. The goal here is still to provide some help to a challenging problem, that has useful applications. While I do get that by having mechanics of gamification and leader-boards with prize money, from a purely game theory point-of-view public LB probing is more and more of a thing; I still believe we shouldn't really encourage this type of practice.
\n\nThat also goes with the fact that purely from a utilitarian point of view, it probably doesn't help much, because of the fact that your private LB has unseen characters, I agree with @haqishen there. If you want to do probing like this, you might as well also just look at the distribution of each root, vowel & consonant in the Bengali language, that at least would be beneficial for those hosting the competition.
\n\nDon't get me wrong, I think those are all very interesting ideas, and I'm pretty amazed at how smart some can be. I just think it's better if we can also have a discussion about it.
", "rawMarkdown": "This might be an unpopular opinion, but I would hope that we could stay away from such practices. The goal here is still to provide some help to a challenging problem, that has useful applications. While I do get that by having mechanics of gamification and leader-boards with prize money, from a purely game theory point-of-view public LB probing is more and more of a thing; I still believe we shouldn't really encourage this type of practice.\n\nThat also goes with the fact that purely from a utilitarian point of view, it probably doesn't help much, because of the fact that your private LB has unseen characters, I agree with @haqishen there. If you want to do probing like this, you might as well also just look at the distribution of each root, vowel & consonant in the Bengali language, that at least would be beneficial for those hosting the competition.\n\nDon't get me wrong, I think those are all very interesting ideas, and I'm pretty amazed at how smart some can be. I just think it's better if we can also have a discussion about it.", "votes": 5 } ] } ], "comments": [ { "id": 723248, "author_name": "Optimo", "author_url": "", "post_date": "2020-01-19T18:19:30.753000", "content": "@bibek777 it might be due to the learning rate, if you have the same for each head. I saw with a basic resnet18 that training with a high learning rate like 0.1 made the vowel_diacritic and consonant_diacritic learn well on first 5 epochs, while grapheme_root was doing worse than random systematically.
Not sure if this is just because of different number of classes or if it's really something.\nNot sure what can be done either : \n- different learning rates for different heads (but if you have just a linear layer in each head it probably won't help much)\n- freeze two heads and fine tune the last one alternatively on each type (vowel, consonant or grapheme - per batch or per epoch maybe) with different learning rates, this seems a bit weird but I wonder if anyone tried it.
", "votes": 1, "replies": [] }, { "id": 720318, "author_name": "Dhananjay Raut", "author_url": "", "post_date": "2020-01-16T10:17:13.213000", "content": "@bibek777 what do you mean by splitting your CV strategy into overall CV and class scores is the way forward
", "votes": 1, "replies": [ { "id": 720323, "author_name": "Bibek", "author_url": "", "post_date": "2020-01-16T10:20:33.330000", "content": "something like this:\n\nOverall CV: XXX\ngrapheme_root : xxx\nvowel_diacritic: yyy\nconsonant_diacritic: zzz\n
\n\n\nLooks like we have more grapheme_root samples
\n
The number of samples of all three components is going to be equal.
\nbut looks like grapheme root part is more difficult to classify in the test set as compared to the training set.
\n\n\nThe number of samples of all three components is going to be equal.
\n
what makes you think that??
", "votes": 0, "replies": [] }, { "id": 720319, "author_name": "Dhananjay Raut", "author_url": "", "post_date": "2020-01-16T10:17:54.647000", "content": "because each test image contains all the three components.
", "votes": 2, "replies": [] }, { "id": 720325, "author_name": "Bibek", "author_url": "", "post_date": "2020-01-16T10:21:34.567000", "content": "Thanx!! I will update my post
", "votes": 0, "replies": [] }, { "id": 720334, "author_name": "Bibek", "author_url": "", "post_date": "2020-01-16T10:28:01.703000", "content": "but this doesn't explain why increase in the scores of grapheme_root leads to decrease in scores of vowel_diacritic and consonant_diacritic, and vice-versa?
@dhananjay3 I am sorry , how do we know that all test images contain all three component ? if consonant_diacritics and vowel_diacritics are 0 that means there are no such component to detect right ? Does that not make it \"Class Not Present\" for some samples ?\ne.g If we create a classifier with vowel_diacritic = 0 and non-zero , will it not become a binary classification problem? Whereas , there is always a grapeheme_root present in an image .
", "votes": 0, "replies": [] }, { "id": 723692, "author_name": "Dhananjay Raut", "author_url": "", "post_date": "2020-01-20T10:42:36.080000", "content": "@phoenix9032 0 does mean that image does not contain consonant/vowel but we still have to predict that it is not there in the image and score for this will be included in the recall. so it is just another class to classify into.
", "votes": 2, "replies": [] } ] }, { "id": 720307, "author_name": "Thomas Di Martino", "author_url": "", "post_date": "2020-01-16T10:08:37.100000", "content": "In such case, do you think that bagging models that would respectively perform their best on either grapheme_root or vowel_diacritic and consonant_diacritic could be a good idea ? I have not tested it myself for now !
that could be a nice thing to try once you run out of ideas. for now I would recommend you to improve your single model for all classes
", "votes": 0, "replies": [] } ] }, { "id": 724870, "author_name": "A/C", "author_url": "", "post_date": "2020-01-21T15:30:25.220000", "content": "@bibek777 Even I am facing the same problem, I think there are few damaged images (improper crop or improper labelling) which decreases our score !!! as mentioned here - https://www.kaggle.com/c/bengaliai-cv19/discussion/126833 by Nihar. I am personally trying different model together for vowel and consonant and a separate model for graphemes
", "votes": 0, "replies": [] }, { "id": 723908, "author_name": "Maxime Lenormand", "author_url": "", "post_date": "2020-01-20T15:41:33.653000", "content": "I have also tried to reduce not the val_loss but the grapheme_root_val_loss which seemed to give some interesting results. I didn't necessarily measure everything correctly (which I need to start getting in the habit of doing though) but it might be worth investigating more in-depth.
I have been seeing something similar. Maybe adding some more layers which are separate would help (I'm guessing you have a single fully connected layer for each output...)?\n
why not have two separate models? \nOne for grapheme and other for the vowel and consonant? (I am pretty confident this would yield good results, but would increase submission time drastically)
", "votes": 0, "replies": [ { "id": 721187, "author_name": "", "author_url": "", "post_date": "2020-01-17T06:38:04.770000", "content": "", "votes": 0, "replies": [] }, { "id": 745433, "author_name": "عثمان", "author_url": "", "post_date": "2020-02-13T20:07:53.163000", "content": "\n\n\nMulti-task learning (MTL) is a subfield of machine learning in which multiple learning tasks are solved at the same time, while exploiting commonalities and differences across tasks. This can result in improved learning efficiency and prediction accuracy for the task-specific models, when compared to training the models separately. Early versions of MTL were called \"hints\".
\n
https://en.wikipedia.org/wiki/Multi-task_learning
\n\nPlus, as you've noted, increase compute, increased train time, increased inference time, increase ram resources, etc.
", "votes": 0, "replies": [] } ] }, { "id": 720721, "author_name": "", "author_url": "", "post_date": "2020-01-16T17:12:21.990000", "content": "", "votes": -1, "replies": [ { "id": 720764, "author_name": "Bibek", "author_url": "", "post_date": "2020-01-16T17:56:10.593000", "content": "\n\n\nyou should be able to prove class distributions of popular combinations to see if they are consistently different from the train set
\n
things are getting interesting now!!!
", "votes": 0, "replies": [] }, { "id": 720782, "author_name": "Sohier Dane", "author_url": "", "post_date": "2020-01-16T18:13:11.513000", "content": "Looks like failed submissions are correctly applied towards daily limits, but aren't properly showing up in the displayed submission count. I'll file a bug report, but unlimited probing should be infeasible.
", "votes": 3, "replies": [] }, { "id": 720786, "author_name": "", "author_url": "", "post_date": "2020-01-16T18:17:44.040000", "content": "", "votes": -1, "replies": [] }, { "id": 720791, "author_name": "Peter", "author_url": "", "post_date": "2020-01-16T18:28:34.577000", "content": "I don't think private probing is possible in this competition. I mean, technically, what you wrote is possible, but without a good model, you don't have the information you'd like to leak. (Or am I missing something?)
", "votes": 0, "replies": [] }, { "id": 720795, "author_name": "", "author_url": "", "post_date": "2020-01-16T18:39:04.067000", "content": "", "votes": 0, "replies": [] }, { "id": 721083, "author_name": "Qishen Ha", "author_url": "", "post_date": "2020-01-17T03:04:06.610000", "content": "I didn't even thought about probing public LB test set.\nThe most important thing is that you might not be able to tell whether private LB test set has a same distribution as public LB test set.\nI think probing public LB can only help you overfitting to public LB and nothing more.
", "votes": 6, "replies": [] }, { "id": 721135, "author_name": "hengck23", "author_url": "", "post_date": "2020-01-17T04:57:31.557000", "content": "probing test dataset is a competition trick. i think it is not required here. i estimated that by using ML methods, 0.99+ is possible for private test set as i suggest using normal ML methods is sufficient.
\n\non a side note, you can take a look at the kaggle google doodle competition on how knowing the number of class instances can improve results (i.e. greedy selection of the top results for k classes)
", "votes": 0, "replies": [] }, { "id": 722607, "author_name": "Maxime Lenormand", "author_url": "", "post_date": "2020-01-18T19:53:55.790000", "content": "This might be an unpopular opinion, but I would hope that we could stay away from such practices. The goal here is still to provide some help to a challenging problem, that has useful applications. While I do get that by having mechanics of gamification and leader-boards with prize money, from a purely game theory point-of-view public LB probing is more and more of a thing; I still believe we shouldn't really encourage this type of practice.
\n\nThat also goes with the fact that purely from a utilitarian point of view, it probably doesn't help much, because of the fact that your private LB has unseen characters, I agree with @haqishen there. If you want to do probing like this, you might as well also just look at the distribution of each root, vowel & consonant in the Bengali language, that at least would be beneficial for those hosting the competition.
\n\nDon't get me wrong, I think those are all very interesting ideas, and I'm pretty amazed at how smart some can be. I just think it's better if we can also have a discussion about it.
", "votes": 5, "replies": [] } ] } ], "raw_markdown_by_id": { "720273": "As I see improvements in my local CV, I also notice a strange tradeoff in the CV scores of `grapheme_root`, `vowel_diacritic` and `consonant_diacritic`. \n\nImprovement in the score of `grapheme_root` leads to relative decrease in the scores of `vowel_diacritic` and `consonant_diacritic`, and vice-versa. So model with high score of `grapheme_root` but with low overall CV may score better on LB compared to the model with high CV and low `grapheme_root` score. This is what exactly happened to me recently, first time when improvement in CV did not correlate to LB increase. \n\nSo make sure you split your CV strategy into overall CV and class scores\n", "723248": "@bibek777 it might be due to the learning rate, if you have the same for each head. I saw with a basic resnet18 that training with a high learning rate like 0.1 made the `vowel_diacritic` and `consonant_diacritic` learn well on first 5 epochs, while `grapheme_root` was doing worse than random systematically.\n\nNot sure if this is just because of different number of classes or if it's really something.\nNot sure what can be done either : \n- different learning rates for different heads (but if you have just a linear layer in each head it probably won't help much)\n- freeze two heads and fine tune the last one alternatively on each type (vowel, consonant or grapheme - per batch or per epoch maybe) with different learning rates, this seems a bit weird but I wonder if anyone tried it.", "720318": "@bibek777 what do you mean by splitting your CV strategy into overall CV and class scores is the way forward", "720315": "> Looks like we have more grapheme_root samples\n\nThe number of samples of all three components is going to be equal. \nbut looks like grapheme root part is more difficult to classify in the test set as compared to the training set.", "720307": "In such case, do you think that bagging models that would respectively perform their best on either `grapheme_root` or `vowel_diacritic` and `consonant_diacritic` could be a good idea ? I have not tested it myself for now !", "724870": "@bibek777 Even I am facing the same problem, I think there are few damaged images (improper crop or improper labelling) which decreases our score !!! as mentioned here - https://www.kaggle.com/c/bengaliai-cv19/discussion/126833 by Nihar. I am personally trying different model together for vowel and consonant and a separate model for graphemes", "723908": "I have also tried to reduce not the `val_loss` but the `grapheme_root_val_loss` which seemed to give some interesting results. I didn't necessarily measure everything correctly (which I need to start getting in the habit of doing though) but it might be worth investigating more in-depth.", "720908": "I have been seeing something similar. Maybe adding some more layers which are separate would help (I'm guessing you have a single fully connected layer for each output...)?\n", "720755": "why not have two separate models? \nOne for grapheme and other for the vowel and consonant? (I am pretty confident this would yield good results, but would increase submission time drastically)", "720721": "" } }