{
  "id": 208326,
  "title": "[TF.Keras]: Implementation of Generalized Cross Entropy Loss",
  "url": "/competitions/cassava-leaf-disease-classification/discussion/208326",
  "author_name": "Innat",
  "post_date": "2021-01-02T23:32:46.125000",
  "votes": 8,
  "comment_count": 0,
  "views": 0,
  "content": "<p>Paper: <a href=\"https://arxiv.org/abs/1805.07836\" target=\"_blank\">Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels</a></p>\n<p>Here is an implementation of it. Hope it helps.</p>\n<pre><code>class GeneralizedCrossEntropy(tf.losses.Loss):\n    def __init__(self, eta=0.7):\n        '''\n        Paper: https://arxiv.org/abs/1805.07836\n        '''\n        super(GeneralizedCrossEntropy, self).__init__()\n        self.eta = eta\n\n    def call(self, y_true, y_pred):\n        t_loss = (1 - tf.pow(tf.reduce_sum(y_true * y_pred, axis=-1), \n                        self.eta)) / self.eta\n        return tf.reduce_mean(t_loss)\n</code></pre>\n<p>Abstract:</p>\n<pre><code>Deep neural networks (DNNs) have achieved tremendous success in a variety of applications across many disciplines. Yet, their superior performance comes with the expensive cost of requiring correctly annotated large-scale datasets. Moreover, due to DNNs' rich capacity, errors in training labels can hamper performance. To combat this problem, mean absolute error (MAE) has recently been proposed as a noise-robust alternative to the commonly-used categorical cross-entropy (CCE) loss. However, as we show in this paper, MAE can perform poorly with DNNs and challenging datasets. Here, we present a theoretically grounded set of noise-robust loss functions that can be seen as a generalization of MAE and CCE. Proposed loss functions can be readily applied with any existing DNN architecture and algorithm while yielding good performance in a wide range of noisy label scenarios. We report results from experiments conducted with CIFAR-10, CIFAR-100, and FASHION-MNIST datasets and synthetically generated noisy labels.\n</code></pre>",
  "messages": [
    {
      "id": 1136293,
      "postDate": "2021-01-02T23:32:46.127Z",
      "content": "<p>Paper: <a href=\"https://arxiv.org/abs/1805.07836\" target=\"_blank\">Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels</a></p>\n<p>Here is an implementation of it. Hope it helps.</p>\n<pre><code>class GeneralizedCrossEntropy(tf.losses.Loss):\n    def __init__(self, eta=0.7):\n        '''\n        Paper: https://arxiv.org/abs/1805.07836\n        '''\n        super(GeneralizedCrossEntropy, self).__init__()\n        self.eta = eta\n\n    def call(self, y_true, y_pred):\n        t_loss = (1 - tf.pow(tf.reduce_sum(y_true * y_pred, axis=-1), \n                        self.eta)) / self.eta\n        return tf.reduce_mean(t_loss)\n</code></pre>\n<p>Abstract:</p>\n<pre><code>Deep neural networks (DNNs) have achieved tremendous success in a variety of applications across many disciplines. Yet, their superior performance comes with the expensive cost of requiring correctly annotated large-scale datasets. Moreover, due to DNNs' rich capacity, errors in training labels can hamper performance. To combat this problem, mean absolute error (MAE) has recently been proposed as a noise-robust alternative to the commonly-used categorical cross-entropy (CCE) loss. However, as we show in this paper, MAE can perform poorly with DNNs and challenging datasets. Here, we present a theoretically grounded set of noise-robust loss functions that can be seen as a generalization of MAE and CCE. Proposed loss functions can be readily applied with any existing DNN architecture and algorithm while yielding good performance in a wide range of noisy label scenarios. We report results from experiments conducted with CIFAR-10, CIFAR-100, and FASHION-MNIST datasets and synthetically generated noisy labels.\n</code></pre>",
      "rawMarkdown": "Paper: [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://arxiv.org/abs/1805.07836)\n\nHere is an implementation of it. Hope it helps.\n\n```python\nclass GeneralizedCrossEntropy(tf.losses.Loss):\n    def __init__(self, eta=0.7):\n        '''\n        Paper: https://arxiv.org/abs/1805.07836\n        '''\n        super(GeneralizedCrossEntropy, self).__init__()\n        self.eta = eta\n\n    def call(self, y_true, y_pred):\n        t_loss = (1 - tf.pow(tf.reduce_sum(y_true * y_pred, axis=-1), \n                        self.eta)) / self.eta\n        return tf.reduce_mean(t_loss)\n```\n\nAbstract:\n\n```\nDeep neural networks (DNNs) have achieved tremendous success in a variety of applications across many disciplines. Yet, their superior performance comes with the expensive cost of requiring correctly annotated large-scale datasets. Moreover, due to DNNs' rich capacity, errors in training labels can hamper performance. To combat this problem, mean absolute error (MAE) has recently been proposed as a noise-robust alternative to the commonly-used categorical cross-entropy (CCE) loss. However, as we show in this paper, MAE can perform poorly with DNNs and challenging datasets. Here, we present a theoretically grounded set of noise-robust loss functions that can be seen as a generalization of MAE and CCE. Proposed loss functions can be readily applied with any existing DNN architecture and algorithm while yielding good performance in a wide range of noisy label scenarios. We report results from experiments conducted with CIFAR-10, CIFAR-100, and FASHION-MNIST datasets and synthetically generated noisy labels.\n```",
      "votes": 7
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "1136293": "Paper: [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://arxiv.org/abs/1805.07836)\n\nHere is an implementation of it. Hope it helps.\n\n```python\nclass GeneralizedCrossEntropy(tf.losses.Loss):\n    def __init__(self, eta=0.7):\n        '''\n        Paper: https://arxiv.org/abs/1805.07836\n        '''\n        super(GeneralizedCrossEntropy, self).__init__()\n        self.eta = eta\n\n    def call(self, y_true, y_pred):\n        t_loss = (1 - tf.pow(tf.reduce_sum(y_true * y_pred, axis=-1), \n                        self.eta)) / self.eta\n        return tf.reduce_mean(t_loss)\n```\n\nAbstract:\n\n```\nDeep neural networks (DNNs) have achieved tremendous success in a variety of applications across many disciplines. Yet, their superior performance comes with the expensive cost of requiring correctly annotated large-scale datasets. Moreover, due to DNNs' rich capacity, errors in training labels can hamper performance. To combat this problem, mean absolute error (MAE) has recently been proposed as a noise-robust alternative to the commonly-used categorical cross-entropy (CCE) loss. However, as we show in this paper, MAE can perform poorly with DNNs and challenging datasets. Here, we present a theoretically grounded set of noise-robust loss functions that can be seen as a generalization of MAE and CCE. Proposed loss functions can be readily applied with any existing DNN architecture and algorithm while yielding good performance in a wide range of noisy label scenarios. We report results from experiments conducted with CIFAR-10, CIFAR-100, and FASHION-MNIST datasets and synthetically generated noisy labels.\n```"
  }
}