Most of public kernels approached this task using models with 3 heads + softmax, @seesee used 1 head + softmax and decoded predictions.
Anyone tried to approach it as multilabel with a 1 head + sigmoid.
What would be pros and/or cons? if any....
", "messages": [ { "id": "765246", "postDate": "03/06/2020 12:16:06", "content": "Most of public kernels approached this task using models with 3 heads + softmax, @seesee used 1 head + softmax and decoded predictions.
Anyone tried to approach it as multilabel with a 1 head + sigmoid.
What would be pros and/or cons? if any....
", "rawMarkdown": "Most of public kernels approached this task using models with `3 heads + softmax`, @seesee used [`1 head + softmax`](https://www.kaggle.com/c/bengaliai-cv19/discussion/134161) and decoded predictions. \n\nAnyone tried to approach it as multilabel with a `1 head + sigmoid`.\n\nWhat would be pros and/or cons? if any....", "votes": null }, { "id": "765252", "postDate": "03/06/2020 12:20:24", "content": "softmax is nothing but a more generalized version of the sigmoid. sigmoid will work only if there are two classes [0,1]. \nAs here minimum class is 7. even if you use one head still you have to go for softmax
", "rawMarkdown": "softmax is nothing but a more generalized version of the sigmoid. sigmoid will work only if there are two classes [0,1]. \nAs here minimum class is 7. even if you use one head still you have to go for softmax", "votes": null }, { "id": "765262", "postDate": "03/06/2020 12:34:52", "content": "Nope, you are wrong. You don't need to go with softmax and here is why.\nWith sigmoid function you predict absence-presence [0, 1] for each of the 186 (168+7+11) classes and each image has three positives.
\n\nI know it can be used. Just wonder what more experienced guys think why this could or couldn't work.
\n\nUPDATED: previous version had wrong number of classes thx @carlosmir
", "rawMarkdown": "Nope, you are wrong. You don't need to go with softmax and here is why.\nWith sigmoid function you predict absence-presence [0, 1] for each of the 186 (168+7+11) classes and each image has three positives.\n\nI know it can be used. Just wonder what more experienced guys think why this could or couldn't work.\n\nUPDATED: previous version had wrong number of classes thx @carlosmir", "votes": null }, { "id": "765265", "postDate": "03/06/2020 12:35:43", "content": "well, in that case, i am not sure
", "rawMarkdown": "well, in that case, i am not sure", "votes": null }, { "id": "765275", "postDate": "03/06/2020 12:53:53", "content": "\nThere are roughly 10,000 possible graphemes, of which roughly 1,000 are represented in the training set. \n\nRead the rules carefully, It means that you don't know exactly how much classes we have.\nBut maximum is 168 * 11 * 7 = 12936 classes.
TL;DR: I got bad results using sigmoid where softmax can be used. Softmax helps convergence.
\n\nHi @valanm \nFrom what I understand of @seesee's work, graphemes are identified by a unique triplet (grapheme_root, vowel_diacritic, consonant_diacritic). Out of 12 936 possible combinations, there are only 1 292 present in the training data set.
> With sigmoid function you predict absence-presence [0, 1] for each of the 1292 classes and each image has three positives.
\n\nI don't exactly get what you mean here, but as @mks2192 suggested, using 1292 classes one can only think of using softmax, since we know there is a single correct answer for each label value. There are no three positives for each image in this setting.
\n\nHowever, I guess there is a way to make it work. From the top of my head, the model would be identifying the three labels in a single vector, i.e. the labels would be 168+11+7=186 sized binary vectors and the model would have to set three of these 186 components to 1.
\n\n(\nJust a little parenthesis. Single-head model === 3-head model. The number of layers and parameters (connections) is the same, i.e. in a 3-head model all the outputs from layer[-2] are connected to head0, head 1 and head 2 (so all the 186 outputs of the model), and in a single-head model all the outputs of layer[-2] are also connected to all the 186 outputs of the model. The 3 heads just allows us to specify groups of outputs. We can achieve the same by treating outputs 0 to 167, 168 to 179 and 180 to 186 independently (applying different losses, for example).\n)
\n\nSo, now, not only has the model to identify three things, but it is not constrained to choose one of each label... But you will have to constrain it :D What I mean by constraining is associating a segment of the output vector to a label (e.g. 0-168 are the first label, etc). And you have to do it in order to train it. Then, you would be back at 3-heads + sigmoid, which makes no sense in this scenario.
\n\nI personally got quite bad results in another task using sigmoid where softmax could be used.
", "rawMarkdown": "TL;DR: I got bad results using sigmoid where softmax can be used. Softmax helps convergence.\n\n---\n\nHi @valanm \nFrom what I understand of @seesee's work, graphemes are identified by a unique triplet `(grapheme_root, vowel_diacritic, consonant_diacritic)`. Out of 12 936 possible combinations, there are only 1 292 present in the training data set.\n\n> With sigmoid function you predict absence-presence [0, 1] for each of the 1292 classes and each image has three positives.\n\nI don't exactly get what you mean here, but as @mks2192 suggested, using 1292 classes one can only think of using softmax, since we know there is a single correct answer for each label value. There are no _three positives_ for each image in this setting.\n\n--- \n\nHowever, I guess there is a way to make it work. From the top of my head, the model would be identifying the three labels in a single vector, i.e. the labels would be 168+11+7=186 sized binary vectors and the model would have to set three of these 186 components to 1.\n\n(\nJust a little parenthesis. Single-head model === 3-head model. The number of layers and parameters (connections) is the same, i.e. in a 3-head model all the outputs from layer[-2] are connected to head0, head 1 and head 2 (so all the 186 outputs of the model), and in a single-head model all the outputs of layer[-2] are also connected to all the 186 outputs of the model. The 3 heads just allows us to specify groups of outputs. We can achieve the same by treating outputs 0 to 167, 168 to 179 and 180 to 186 independently (applying different losses, for example).\n)\n\nSo, now, not only has the model to identify three things, but it is not constrained to choose one of each label... But you will have to constrain it :D What I mean by constraining is associating a segment of the output vector to a label (e.g. 0-168 are the first label, etc). And you have to do it in order to train it. Then, you would be back at 3-heads + sigmoid, which makes no sense in this scenario.\n\nI personally got quite bad results in another task using sigmoid where softmax could be used.", "votes": null }, { "id": "765432", "postDate": "03/06/2020 16:07:08", "content": "From 186 sigmoid outputs you take max(0-167), max(7) and max(11) and you end up at the same place as with 3 heads + softmax.
", "rawMarkdown": "From 186 sigmoid outputs you take max(0-167), max(7) and max(11) and you end up at the same place as with 3 heads + softmax.", "votes": null } ], "comments": [ { "id": 765252, "author_name": "mks2192", "author_url": "", "post_date": "03/06/2020 12:20:24", "content": "softmax is nothing but a more generalized version of the sigmoid. sigmoid will work only if there are two classes [0,1]. \nAs here minimum class is 7. even if you use one head still you have to go for softmax
", "votes": null, "replies": [ { "id": 765262, "author_name": "valanm", "author_url": "", "post_date": "03/06/2020 12:34:52", "content": "Nope, you are wrong. You don't need to go with softmax and here is why.\nWith sigmoid function you predict absence-presence [0, 1] for each of the 186 (168+7+11) classes and each image has three positives.
\n\nI know it can be used. Just wonder what more experienced guys think why this could or couldn't work.
\n\nUPDATED: previous version had wrong number of classes thx @carlosmir
", "votes": null, "replies": [] }, { "id": 765265, "author_name": "mks2192", "author_url": "", "post_date": "03/06/2020 12:35:43", "content": "well, in that case, i am not sure
", "votes": null, "replies": [] }, { "id": 765275, "author_name": "smivvla", "author_url": "", "post_date": "03/06/2020 12:53:53", "content": "\nThere are roughly 10,000 possible graphemes, of which roughly 1,000 are represented in the training set. \n\nRead the rules carefully, It means that you don't know exactly how much classes we have.\nBut maximum is 168 * 11 * 7 = 12936 classes.
TL;DR: I got bad results using sigmoid where softmax can be used. Softmax helps convergence.
\n\nHi @valanm \nFrom what I understand of @seesee's work, graphemes are identified by a unique triplet (grapheme_root, vowel_diacritic, consonant_diacritic). Out of 12 936 possible combinations, there are only 1 292 present in the training data set.
> With sigmoid function you predict absence-presence [0, 1] for each of the 1292 classes and each image has three positives.
\n\nI don't exactly get what you mean here, but as @mks2192 suggested, using 1292 classes one can only think of using softmax, since we know there is a single correct answer for each label value. There are no three positives for each image in this setting.
\n\nHowever, I guess there is a way to make it work. From the top of my head, the model would be identifying the three labels in a single vector, i.e. the labels would be 168+11+7=186 sized binary vectors and the model would have to set three of these 186 components to 1.
\n\n(\nJust a little parenthesis. Single-head model === 3-head model. The number of layers and parameters (connections) is the same, i.e. in a 3-head model all the outputs from layer[-2] are connected to head0, head 1 and head 2 (so all the 186 outputs of the model), and in a single-head model all the outputs of layer[-2] are also connected to all the 186 outputs of the model. The 3 heads just allows us to specify groups of outputs. We can achieve the same by treating outputs 0 to 167, 168 to 179 and 180 to 186 independently (applying different losses, for example).\n)
\n\nSo, now, not only has the model to identify three things, but it is not constrained to choose one of each label... But you will have to constrain it :D What I mean by constraining is associating a segment of the output vector to a label (e.g. 0-168 are the first label, etc). And you have to do it in order to train it. Then, you would be back at 3-heads + sigmoid, which makes no sense in this scenario.
\n\nI personally got quite bad results in another task using sigmoid where softmax could be used.
", "votes": null, "replies": [] }, { "id": 765432, "author_name": "valanm", "author_url": "", "post_date": "03/06/2020 16:07:08", "content": "From 186 sigmoid outputs you take max(0-167), max(7) and max(11) and you end up at the same place as with 3 heads + softmax.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "765246": "Most of public kernels approached this task using models with `3 heads + softmax`, @seesee used [`1 head + softmax`](https://www.kaggle.com/c/bengaliai-cv19/discussion/134161) and decoded predictions. \n\nAnyone tried to approach it as multilabel with a `1 head + sigmoid`.\n\nWhat would be pros and/or cons? if any....", "765252": "softmax is nothing but a more generalized version of the sigmoid. sigmoid will work only if there are two classes [0,1]. \nAs here minimum class is 7. even if you use one head still you have to go for softmax", "765262": "Nope, you are wrong. You don't need to go with softmax and here is why.\nWith sigmoid function you predict absence-presence [0, 1] for each of the 186 (168+7+11) classes and each image has three positives.\n\nI know it can be used. Just wonder what more experienced guys think why this could or couldn't work.\n\nUPDATED: previous version had wrong number of classes thx @carlosmir", "765265": "well, in that case, i am not sure", "765275": "```\nThere are roughly 10,000 possible graphemes, of which roughly 1,000 are represented in the training set. \n```\nRead the rules carefully, It means that you don't know exactly how much classes we have.\nBut maximum is 168 * 11 * 7 = 12936 classes.", "765414": "TL;DR: I got bad results using sigmoid where softmax can be used. Softmax helps convergence.\n\n---\n\nHi @valanm \nFrom what I understand of @seesee's work, graphemes are identified by a unique triplet `(grapheme_root, vowel_diacritic, consonant_diacritic)`. Out of 12 936 possible combinations, there are only 1 292 present in the training data set.\n\n> With sigmoid function you predict absence-presence [0, 1] for each of the 1292 classes and each image has three positives.\n\nI don't exactly get what you mean here, but as @mks2192 suggested, using 1292 classes one can only think of using softmax, since we know there is a single correct answer for each label value. There are no _three positives_ for each image in this setting.\n\n--- \n\nHowever, I guess there is a way to make it work. From the top of my head, the model would be identifying the three labels in a single vector, i.e. the labels would be 168+11+7=186 sized binary vectors and the model would have to set three of these 186 components to 1.\n\n(\nJust a little parenthesis. Single-head model === 3-head model. The number of layers and parameters (connections) is the same, i.e. in a 3-head model all the outputs from layer[-2] are connected to head0, head 1 and head 2 (so all the 186 outputs of the model), and in a single-head model all the outputs of layer[-2] are also connected to all the 186 outputs of the model. The 3 heads just allows us to specify groups of outputs. We can achieve the same by treating outputs 0 to 167, 168 to 179 and 180 to 186 independently (applying different losses, for example).\n)\n\nSo, now, not only has the model to identify three things, but it is not constrained to choose one of each label... But you will have to constrain it :D What I mean by constraining is associating a segment of the output vector to a label (e.g. 0-168 are the first label, etc). And you have to do it in order to train it. Then, you would be back at 3-heads + sigmoid, which makes no sense in this scenario.\n\nI personally got quite bad results in another task using sigmoid where softmax could be used.", "765432": "From 186 sigmoid outputs you take max(0-167), max(7) and max(11) and you end up at the same place as with 3 heads + softmax." }, "source": "meta" }