{
  "id": 208324,
  "title": "[TF.Keras] Implementation of Symmetric Cross Entropy Loss",
  "url": "/competitions/cassava-leaf-disease-classification/discussion/208324",
  "author_name": "",
  "post_date": "2021-01-02T23:24:16.933781900Z",
  "votes": 17,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Paper: <a href=\"https://arxiv.org/abs/1908.06112\" target=\"_blank\">Symmetric Cross-Entropy for Robust Learning with Noisy Labels</a></p>\n<p>Below is an Implementation of this special cross-entropy loss function for noisy labels.</p>\n<pre><code>class SymmetricCrossEntropy(tf.losses.Loss):\n    def __init__(self, alpha=0.1, beta=1.0):\n        '''\n        Paper: https://arxiv.org/abs/1908.06112\n        '''\n        super(SymmetricCrossEntropy, self).__init__()\n        self.alpha = alpha\n        self.beta = beta\n\n    def call(self, y_true, y_pred):\n        ce_loss = tf.reduce_mean(-tf.reduce_sum(y_true * \\\n                    tf.math.log(tf.clip_by_value(y_pred, 1e-7, 1.0)), \n                    axis = -1))\n\n        rce_loss = tf.reduce_mean(-tf.reduce_sum(y_pred * \\\n                   tf.math.log(tf.clip_by_value(y_true, 1e-4, 1.0)), \n                   axis = -1))\n\n        return self.alpha*ce_loss + self.beta*rce_loss\n</code></pre>",
  "messages": [
    {
      "id": "1136285",
      "postDate": "01/02/2021 23:24:16",
      "content": "<p>Paper: <a href=\"https://arxiv.org/abs/1908.06112\" target=\"_blank\">Symmetric Cross-Entropy for Robust Learning with Noisy Labels</a></p>\n<p>Below is an Implementation of this special cross-entropy loss function for noisy labels.</p>\n<pre><code>class SymmetricCrossEntropy(tf.losses.Loss):\n    def __init__(self, alpha=0.1, beta=1.0):\n        '''\n        Paper: https://arxiv.org/abs/1908.06112\n        '''\n        super(SymmetricCrossEntropy, self).__init__()\n        self.alpha = alpha\n        self.beta = beta\n\n    def call(self, y_true, y_pred):\n        ce_loss = tf.reduce_mean(-tf.reduce_sum(y_true * \\\n                    tf.math.log(tf.clip_by_value(y_pred, 1e-7, 1.0)), \n                    axis = -1))\n\n        rce_loss = tf.reduce_mean(-tf.reduce_sum(y_pred * \\\n                   tf.math.log(tf.clip_by_value(y_true, 1e-4, 1.0)), \n                   axis = -1))\n\n        return self.alpha*ce_loss + self.beta*rce_loss\n</code></pre>",
      "rawMarkdown": "Paper: [Symmetric Cross-Entropy for Robust Learning with Noisy Labels](https://arxiv.org/abs/1908.06112)\n\nBelow is an Implementation of this special cross-entropy loss function for noisy labels.\n\n```python\nclass SymmetricCrossEntropy(tf.losses.Loss):\n    def __init__(self, alpha=0.1, beta=1.0):\n        '''\n        Paper: https://arxiv.org/abs/1908.06112\n        '''\n        super(SymmetricCrossEntropy, self).__init__()\n        self.alpha = alpha\n        self.beta = beta\n\n    def call(self, y_true, y_pred):\n        ce_loss = tf.reduce_mean(-tf.reduce_sum(y_true * \\\n                    tf.math.log(tf.clip_by_value(y_pred, 1e-7, 1.0)), \n                    axis = -1))\n        \n        rce_loss = tf.reduce_mean(-tf.reduce_sum(y_pred * \\\n                   tf.math.log(tf.clip_by_value(y_true, 1e-4, 1.0)), \n                   axis = -1))\n        \n        return self.alpha*ce_loss + self.beta*rce_loss\n```",
      "votes": null
    },
    {
      "id": "1149116",
      "postDate": "01/11/2021 15:47:07",
      "content": "<p>I tried to use it. But the network is not learning anything accuracy is below 30 after 10 epochs. I even tried increasing LR but not working. Also, I have used the same Alpha and Beta parameters as given in the paper.</p>",
      "rawMarkdown": "I tried to use it. But the network is not learning anything accuracy is below 30 after 10 epochs. I even tried increasing LR but not working. Also, I have used the same Alpha and Beta parameters as given in the paper.",
      "votes": null
    },
    {
      "id": "1149482",
      "postDate": "01/11/2021 21:37:36",
      "content": "<p>Are you saying the accuracy of your model dropped below 30 after epoch 10? Or, it's from the beginning? </p>",
      "rawMarkdown": "Are you saying the accuracy of your model dropped below 30 after epoch 10? Or, it's from the beginning?",
      "votes": null
    },
    {
      "id": "1149528",
      "postDate": "01/11/2021 22:59:25",
      "content": "<p>I think there's some issue in your training set up. FYI, here is a baseline model score with this loss function in my set up. </p>\n<pre><code>[INFO]: OOF Fold No. 0\n[INFO]: Model Build Successfully.\n[INFO]: Model EfficientNetB0 - Image Size 224 - Batch Size 64 - LR: 0.001\nTotal params: 4,049,564\nTrainable params: 4,007,548\nNon-trainable params: 42,016\n\nEpoch 1/5 4s/step - loss: 3.3497 - categorical_accuracy: 0.6680 - val_loss: 3.4368 - val_categorical_accuracy: 0.6664\nEpoch 2/5 3s/step - loss: 2.8629 - categorical_accuracy: 0.7151 - val_loss: 2.5283 - val_categorical_accuracy: 0.7535\nEpoch 3/5 3s/step - loss: 2.6401 - categorical_accuracy: 0.7393 - val_loss: 3.2577 - val_categorical_accuracy: 0.6820\n\nEpoch 00004: ReduceLROnPlateau reducing learning rate to 0.0003000000142492354.\nEpoch 4/5 2s/step - loss: 2.4805 - categorical_accuracy: 0.7519 - val_loss: 2.5751 - val_categorical_accuracy: 0.7458\nEpoch 5/5 2s/step - loss: 2.2050 - categorical_accuracy: 0.7793 - val_loss: 1.9150 - val_categorical_accuracy: 0.8079\n</code></pre>",
      "rawMarkdown": "I think there's some issue in your training set up. FYI, here is a baseline model score with this loss function in my set up. \n\n\n```\n[INFO]: OOF Fold No. 0\n[INFO]: Model Build Successfully.\n[INFO]: Model EfficientNetB0 - Image Size 224 - Batch Size 64 - LR: 0.001\nTotal params: 4,049,564\nTrainable params: 4,007,548\nNon-trainable params: 42,016\n\nEpoch 1/5 4s/step - loss: 3.3497 - categorical_accuracy: 0.6680 - val_loss: 3.4368 - val_categorical_accuracy: 0.6664\nEpoch 2/5 3s/step - loss: 2.8629 - categorical_accuracy: 0.7151 - val_loss: 2.5283 - val_categorical_accuracy: 0.7535\nEpoch 3/5 3s/step - loss: 2.6401 - categorical_accuracy: 0.7393 - val_loss: 3.2577 - val_categorical_accuracy: 0.6820\n\nEpoch 00004: ReduceLROnPlateau reducing learning rate to 0.0003000000142492354.\nEpoch 4/5 2s/step - loss: 2.4805 - categorical_accuracy: 0.7519 - val_loss: 2.5751 - val_categorical_accuracy: 0.7458\nEpoch 5/5 2s/step - loss: 2.2050 - categorical_accuracy: 0.7793 - val_loss: 1.9150 - val_categorical_accuracy: 0.8079\n```",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1149116,
      "author_name": "prateek0x",
      "author_url": "",
      "post_date": "01/11/2021 15:47:07",
      "content": "<p>I tried to use it. But the network is not learning anything accuracy is below 30 after 10 epochs. I even tried increasing LR but not working. Also, I have used the same Alpha and Beta parameters as given in the paper.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1149482,
          "author_name": "ipythonx",
          "author_url": "",
          "post_date": "01/11/2021 21:37:36",
          "content": "<p>Are you saying the accuracy of your model dropped below 30 after epoch 10? Or, it's from the beginning? </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1149528,
          "author_name": "ipythonx",
          "author_url": "",
          "post_date": "01/11/2021 22:59:25",
          "content": "<p>I think there's some issue in your training set up. FYI, here is a baseline model score with this loss function in my set up. </p>\n<pre><code>[INFO]: OOF Fold No. 0\n[INFO]: Model Build Successfully.\n[INFO]: Model EfficientNetB0 - Image Size 224 - Batch Size 64 - LR: 0.001\nTotal params: 4,049,564\nTrainable params: 4,007,548\nNon-trainable params: 42,016\n\nEpoch 1/5 4s/step - loss: 3.3497 - categorical_accuracy: 0.6680 - val_loss: 3.4368 - val_categorical_accuracy: 0.6664\nEpoch 2/5 3s/step - loss: 2.8629 - categorical_accuracy: 0.7151 - val_loss: 2.5283 - val_categorical_accuracy: 0.7535\nEpoch 3/5 3s/step - loss: 2.6401 - categorical_accuracy: 0.7393 - val_loss: 3.2577 - val_categorical_accuracy: 0.6820\n\nEpoch 00004: ReduceLROnPlateau reducing learning rate to 0.0003000000142492354.\nEpoch 4/5 2s/step - loss: 2.4805 - categorical_accuracy: 0.7519 - val_loss: 2.5751 - val_categorical_accuracy: 0.7458\nEpoch 5/5 2s/step - loss: 2.2050 - categorical_accuracy: 0.7793 - val_loss: 1.9150 - val_categorical_accuracy: 0.8079\n</code></pre>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1136285": "Paper: [Symmetric Cross-Entropy for Robust Learning with Noisy Labels](https://arxiv.org/abs/1908.06112)\n\nBelow is an Implementation of this special cross-entropy loss function for noisy labels.\n\n```python\nclass SymmetricCrossEntropy(tf.losses.Loss):\n    def __init__(self, alpha=0.1, beta=1.0):\n        '''\n        Paper: https://arxiv.org/abs/1908.06112\n        '''\n        super(SymmetricCrossEntropy, self).__init__()\n        self.alpha = alpha\n        self.beta = beta\n\n    def call(self, y_true, y_pred):\n        ce_loss = tf.reduce_mean(-tf.reduce_sum(y_true * \\\n                    tf.math.log(tf.clip_by_value(y_pred, 1e-7, 1.0)), \n                    axis = -1))\n        \n        rce_loss = tf.reduce_mean(-tf.reduce_sum(y_pred * \\\n                   tf.math.log(tf.clip_by_value(y_true, 1e-4, 1.0)), \n                   axis = -1))\n        \n        return self.alpha*ce_loss + self.beta*rce_loss\n```",
    "1149116": "I tried to use it. But the network is not learning anything accuracy is below 30 after 10 epochs. I even tried increasing LR but not working. Also, I have used the same Alpha and Beta parameters as given in the paper.",
    "1149482": "Are you saying the accuracy of your model dropped below 30 after epoch 10? Or, it's from the beginning?",
    "1149528": "I think there's some issue in your training set up. FYI, here is a baseline model score with this loss function in my set up. \n\n\n```\n[INFO]: OOF Fold No. 0\n[INFO]: Model Build Successfully.\n[INFO]: Model EfficientNetB0 - Image Size 224 - Batch Size 64 - LR: 0.001\nTotal params: 4,049,564\nTrainable params: 4,007,548\nNon-trainable params: 42,016\n\nEpoch 1/5 4s/step - loss: 3.3497 - categorical_accuracy: 0.6680 - val_loss: 3.4368 - val_categorical_accuracy: 0.6664\nEpoch 2/5 3s/step - loss: 2.8629 - categorical_accuracy: 0.7151 - val_loss: 2.5283 - val_categorical_accuracy: 0.7535\nEpoch 3/5 3s/step - loss: 2.6401 - categorical_accuracy: 0.7393 - val_loss: 3.2577 - val_categorical_accuracy: 0.6820\n\nEpoch 00004: ReduceLROnPlateau reducing learning rate to 0.0003000000142492354.\nEpoch 4/5 2s/step - loss: 2.4805 - categorical_accuracy: 0.7519 - val_loss: 2.5751 - val_categorical_accuracy: 0.7458\nEpoch 5/5 2s/step - loss: 2.2050 - categorical_accuracy: 0.7793 - val_loss: 1.9150 - val_categorical_accuracy: 0.8079\n```"
  },
  "source": "meta"
}