{
  "id": 213067,
  "title": "Loss function & Activations",
  "url": "/competitions/ranzcr-clip-catheter-line-classification/discussion/213067",
  "author_name": "",
  "post_date": "2021-01-21T11:20:36.283785600Z",
  "votes": 1,
  "comment_count": 5,
  "views": 0,
  "content": "<p>I've noticed people seem to use binary cross entropy instead of categorical cross entropy and sigmoidal instead of softmax activation. Why do these work better if we have a multilabel classification problem?</p>",
  "messages": [
    {
      "id": "1162831",
      "postDate": "01/21/2021 11:20:36",
      "content": "<p>I've noticed people seem to use binary cross entropy instead of categorical cross entropy and sigmoidal instead of softmax activation. Why do these work better if we have a multilabel classification problem?</p>",
      "rawMarkdown": "I've noticed people seem to use binary cross entropy instead of categorical cross entropy and sigmoidal instead of softmax activation. Why do these work better if we have a multilabel classification problem?",
      "votes": null
    },
    {
      "id": "1162872",
      "postDate": "01/21/2021 11:41:37",
      "content": "<p>because the sum of all label for each sample does not equal to 1</p>",
      "rawMarkdown": "because the sum of all label for each sample does not equal to 1",
      "votes": null
    },
    {
      "id": "1169093",
      "postDate": "01/25/2021 10:09:37",
      "content": "<p>Because we have not a multilabel classification problem, but 11 binary classification problems for which there is a single optimizer (AUC). Therefore, the activation function for binary classification is used.</p>",
      "rawMarkdown": "Because we have not a multilabel classification problem, but 11 binary classification problems for which there is a single optimizer (AUC). Therefore, the activation function for binary classification is used.",
      "votes": null
    },
    {
      "id": "1174451",
      "postDate": "01/28/2021 13:35:41",
      "content": "<p>Spot on from Maksym why you should think of this as multiple binary classification task (however, with some correlation of the tasks - plus a few combinations might not be possible). Of course, the options you mentioned are not the only ones. We can also try to use various other options for binary classification (e.g. focal loss) or something <a href=\"https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/204690\" target=\"_blank\">more closely aligned with the competition metric</a>.</p>",
      "rawMarkdown": "Spot on from Maksym why you should think of this as multiple binary classification task (however, with some correlation of the tasks - plus a few combinations might not be possible). Of course, the options you mentioned are not the only ones. We can also try to use various other options for binary classification (e.g. focal loss) or something [more closely aligned with the competition metric](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/204690).",
      "votes": null
    },
    {
      "id": "1174464",
      "postDate": "01/28/2021 13:45:56",
      "content": "<p>Actually, we are having multilabel classification problem and we would use softmax if we would have multiclass problem. </p>",
      "rawMarkdown": "Actually, we are having multilabel classification problem and we would use softmax if we would have multiclass problem.",
      "votes": null
    },
    {
      "id": "1175988",
      "postDate": "01/29/2021 12:42:08",
      "content": "<p>It is multilabel classification,  For eg, we have a chest X-ray diseases classification problem in which we have three labels <br>\n[ cardiomegaly, pneumonia, fibrosis]</p>\n<p><strong>Use of Softmax Activation:</strong></p>\n<p><strong>Suppose we have 3 images in the dataset and each image(patient) has one disease</strong></p>\n<p>image1 -&gt; [1,0,0]<br>\nimage2 -&gt; [1,0,0]<br>\nimage3 -&gt; [0,1,0]</p>\n<p>In this case, we want to classify only one disease and <strong>the sum of all labels for each sample will be equal to one.</strong></p>\n<p><strong>Use of Sigmoid Activation for multilabel classification:</strong></p>\n<p><strong>Suppose we have 3 images in the dataset and it is possible that each image(patient) has multiple diseases</strong></p>\n<p>image1 -&gt; [1,1,0]<br>\nimage2 -&gt; [1,0,1]<br>\nimage3 -&gt; [1,1,1]</p>\n<p>In this case, we want to classify multiple diseases and <strong>the sum of all labels for each sample will not be equal to one.</strong></p>\n<p>If you would like to work on this type of problem you can refer to my notebook on NIH multi-label 14- diseases Chest x-ray classification : <br>\n<a href=\"https://www.kaggle.com/parthdhameliya77/nih-multi-label-chest-x-ray-classification\" target=\"_blank\">https://www.kaggle.com/parthdhameliya77/nih-multi-label-chest-x-ray-classification</a></p>",
      "rawMarkdown": "It is multilabel classification,  For eg, we have a chest X-ray diseases classification problem in which we have three labels \n[ cardiomegaly, pneumonia, fibrosis]\n\n**Use of Softmax Activation:**\n\n**Suppose we have 3 images in the dataset and each image(patient) has one disease**\n\nimage1 -> [1,0,0]\nimage2 -> [1,0,0]\nimage3 -> [0,1,0]\n\nIn this case, we want to classify only one disease and **the sum of all labels for each sample will be equal to one.**\n\n**Use of Sigmoid Activation for multilabel classification:**\n\n**Suppose we have 3 images in the dataset and it is possible that each image(patient) has multiple diseases**\n\nimage1 -> [1,1,0]\nimage2 -> [1,0,1]\nimage3 -> [1,1,1]\n\nIn this case, we want to classify multiple diseases and **the sum of all labels for each sample will not be equal to one.**\n\n\nIf you would like to work on this type of problem you can refer to my notebook on NIH multi-label 14- diseases Chest x-ray classification : \nhttps://www.kaggle.com/parthdhameliya77/nih-multi-label-chest-x-ray-classification",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1162872,
      "author_name": "moewie94",
      "author_url": "",
      "post_date": "01/21/2021 11:41:37",
      "content": "<p>because the sum of all label for each sample does not equal to 1</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1169093,
      "author_name": "maksymshkliarevskyi",
      "author_url": "",
      "post_date": "01/25/2021 10:09:37",
      "content": "<p>Because we have not a multilabel classification problem, but 11 binary classification problems for which there is a single optimizer (AUC). Therefore, the activation function for binary classification is used.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1174464,
          "author_name": "opanichev",
          "author_url": "",
          "post_date": "01/28/2021 13:45:56",
          "content": "<p>Actually, we are having multilabel classification problem and we would use softmax if we would have multiclass problem. </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1174451,
      "author_name": "bjoernholzhauer",
      "author_url": "",
      "post_date": "01/28/2021 13:35:41",
      "content": "<p>Spot on from Maksym why you should think of this as multiple binary classification task (however, with some correlation of the tasks - plus a few combinations might not be possible). Of course, the options you mentioned are not the only ones. We can also try to use various other options for binary classification (e.g. focal loss) or something <a href=\"https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/204690\" target=\"_blank\">more closely aligned with the competition metric</a>.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1175988,
      "author_name": "parthdhameliya77",
      "author_url": "",
      "post_date": "01/29/2021 12:42:08",
      "content": "<p>It is multilabel classification,  For eg, we have a chest X-ray diseases classification problem in which we have three labels <br>\n[ cardiomegaly, pneumonia, fibrosis]</p>\n<p><strong>Use of Softmax Activation:</strong></p>\n<p><strong>Suppose we have 3 images in the dataset and each image(patient) has one disease</strong></p>\n<p>image1 -&gt; [1,0,0]<br>\nimage2 -&gt; [1,0,0]<br>\nimage3 -&gt; [0,1,0]</p>\n<p>In this case, we want to classify only one disease and <strong>the sum of all labels for each sample will be equal to one.</strong></p>\n<p><strong>Use of Sigmoid Activation for multilabel classification:</strong></p>\n<p><strong>Suppose we have 3 images in the dataset and it is possible that each image(patient) has multiple diseases</strong></p>\n<p>image1 -&gt; [1,1,0]<br>\nimage2 -&gt; [1,0,1]<br>\nimage3 -&gt; [1,1,1]</p>\n<p>In this case, we want to classify multiple diseases and <strong>the sum of all labels for each sample will not be equal to one.</strong></p>\n<p>If you would like to work on this type of problem you can refer to my notebook on NIH multi-label 14- diseases Chest x-ray classification : <br>\n<a href=\"https://www.kaggle.com/parthdhameliya77/nih-multi-label-chest-x-ray-classification\" target=\"_blank\">https://www.kaggle.com/parthdhameliya77/nih-multi-label-chest-x-ray-classification</a></p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1162831": "I've noticed people seem to use binary cross entropy instead of categorical cross entropy and sigmoidal instead of softmax activation. Why do these work better if we have a multilabel classification problem?",
    "1162872": "because the sum of all label for each sample does not equal to 1",
    "1169093": "Because we have not a multilabel classification problem, but 11 binary classification problems for which there is a single optimizer (AUC). Therefore, the activation function for binary classification is used.",
    "1174451": "Spot on from Maksym why you should think of this as multiple binary classification task (however, with some correlation of the tasks - plus a few combinations might not be possible). Of course, the options you mentioned are not the only ones. We can also try to use various other options for binary classification (e.g. focal loss) or something [more closely aligned with the competition metric](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/204690).",
    "1174464": "Actually, we are having multilabel classification problem and we would use softmax if we would have multiclass problem.",
    "1175988": "It is multilabel classification,  For eg, we have a chest X-ray diseases classification problem in which we have three labels \n[ cardiomegaly, pneumonia, fibrosis]\n\n**Use of Softmax Activation:**\n\n**Suppose we have 3 images in the dataset and each image(patient) has one disease**\n\nimage1 -> [1,0,0]\nimage2 -> [1,0,0]\nimage3 -> [0,1,0]\n\nIn this case, we want to classify only one disease and **the sum of all labels for each sample will be equal to one.**\n\n**Use of Sigmoid Activation for multilabel classification:**\n\n**Suppose we have 3 images in the dataset and it is possible that each image(patient) has multiple diseases**\n\nimage1 -> [1,1,0]\nimage2 -> [1,0,1]\nimage3 -> [1,1,1]\n\nIn this case, we want to classify multiple diseases and **the sum of all labels for each sample will not be equal to one.**\n\n\nIf you would like to work on this type of problem you can refer to my notebook on NIH multi-label 14- diseases Chest x-ray classification : \nhttps://www.kaggle.com/parthdhameliya77/nih-multi-label-chest-x-ray-classification"
  },
  "source": "meta"
}