{"cells":[{"metadata":{},"cell_type":"markdown","source":"A collection of loss function resources that help with noisy labels"},{"metadata":{},"cell_type":"markdown","source":"# Imports"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt # For plotting\n\nfrom keras import backend as K # Losses\nimport tensorflow as tf # Tensorflow","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Standard Loss\n\nThe response of a Log Loss when the target is 1"},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"K.binary_crossentropy","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"inputs = [x/100 for x in range(0, 100, 1)]\noutputs = []\n\nfor i in inputs:\n    outputs.append(K.binary_crossentropy(tf.convert_to_tensor([1.0]), tf.convert_to_tensor([i])).numpy()[0])\n    \nplt.figure(figsize=(20,10))\nplt.plot(inputs, outputs)\nplt.title(\"Response of Log Loss for Target = 1\")\nplt.xlabel(\"Prediction from Model\")\nplt.ylabel(\"Loss Value\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Symmetric Cross Entropy\n\nInspired by the symmetric KL-divergence, the paper proposes the approach of Symmetric cross entropy Learning (SL), boosting CE symmetrically with a noise robust counterpart Reverse Cross Entropy (RCE). The proposed SL simultaneously addresses both the under learning and overfitting problem of CE in the presence of noisy labels\n\nhttps://arxiv.org/pdf/1908.06112.pdf"},{"metadata":{"trusted":true},"cell_type":"code","source":"def symmetric_cross_entropy(y_true, y_pred, alpha, beta):\n    \"\"\"\n    2019 - Symmetric Cross Entropy for Robust Learning with Noisy Labels - https://arxiv.org/pdf/1908.06112.pdf\n    \"\"\"\n    y_true_1 = y_true\n    y_pred_1 = y_pred\n\n    y_true_2 = y_true\n    y_pred_2 = y_pred\n\n    y_pred_1 = tf.clip_by_value(y_pred_1, 1e-7, 1.0)\n    y_true_2 = tf.clip_by_value(y_true_2, 1e-4, 1.0)\n\n    return alpha*tf.reduce_mean(-tf.reduce_sum(y_true_1 * tf.math.log(y_pred_1), axis = -1)) + beta*tf.reduce_mean(-tf.reduce_sum(y_pred_2 * tf.math.log(y_true_2), axis = -1))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"inputs = [x/100 for x in range(0, 100, 1)]\n\nplt.figure(figsize=(20,10))\n\nfor a in [alpha/100 for alpha in range(1, 100, 25)]:\n    outputs = []\n    for i in inputs:\n        outputs.append(symmetric_cross_entropy(tf.convert_to_tensor([1.0]), tf.convert_to_tensor([i]), a, 1.0).numpy())\n    \n    plt.plot(inputs, outputs, label=\"Alpha = \" + str(a) + \"  |  Beta = 1.0\")\n    plt.title(\"Symmetric Cross Entropy Loss for Target = 1\")\n    plt.xlabel(\"Prediction from Model\")\n    plt.ylabel(\"Loss Value\")\n\nplt.legend()\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Label Smoothing Regularization\n\n\nLabel Smoothing Regularizing is done on neural networks by penalizing low entropy output distributions. It prevents overfitting on noisy labels by smoothing labels.\n\nhttps://arxiv.org/pdf/1701.06548.pdf"},{"metadata":{"trusted":true},"cell_type":"code","source":"def lsr(y_true, y_pred, epsilon):\n    \"\"\"\n    2017 - REGULARIZING NEURAL NETWORKS BY PENALIZING CONFIDENT OUTPUT DISTRIBUTIONS - https://arxiv.org/pdf/1701.06548.pdf\n    \"\"\"\n    \n    y_smoothed_true = y_true * (1 - epsilon - epsilon / 10.0)\n    y_smoothed_true = y_smoothed_true + epsilon / 10.0\n\n    y_pred_1 = tf.clip_by_value(y_pred, 1e-7, 1.0)\n\n    return tf.reduce_mean(-tf.reduce_sum(y_smoothed_true * tf.math.log(y_pred_1), axis=-1))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"inputs = [x/100 for x in range(0, 100, 1)]\n\nplt.figure(figsize=(20,10))\n\nfor e in [eps/100 for eps in range(1, 100, 25)]:\n    outputs = []\n    for i in inputs:\n        outputs.append(lsr(tf.convert_to_tensor(1.0), tf.convert_to_tensor([i]), e).numpy())\n    \n    plt.plot(inputs, outputs, label=\"Epsilon = \" + str(e))\n    plt.title(\"Label Smoothing Regularization for Target = 1\")\n    plt.xlabel(\"Prediction from Model\")\n    plt.ylabel(\"Loss Value\")\n\nplt.legend()\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Generalized Cross Entropy Loss\n\nIt is proposed as a theoretically grounded noise-robust loss function that can be seen as a generalization of MAE and CCE.\n\nhttps://arxiv.org/pdf/1805.07836.pdf"},{"metadata":{"trusted":true},"cell_type":"code","source":"def generalized_cross_entropy(y_true, y_pred, q):\n    \"\"\"\n    2018 - Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels - https://arxiv.org/pdf/1805.07836.pdf\n    \"\"\"\n    intermed_ = tf.pow(tf.reduce_sum(y_true * y_pred, axis=-1), q)\n    t_loss = (1 - intermed_) / q\n    return tf.reduce_mean(t_loss)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"inputs = [x/100 for x in range(0, 100, 1)]\n\nplt.figure(figsize=(20,10))\n\nfor q in [limit/100 for limit in range(10, 150, 25)]:\n    outputs = []\n    for i in inputs:\n        outputs.append(generalized_cross_entropy(tf.convert_to_tensor([1.0]), tf.convert_to_tensor([i]), q).numpy())\n    \n    plt.plot(inputs, outputs, label=\"q = \" + str(q))\n    plt.title(\"Generalized Cross Entropy Loss for Target = 1\")\n    plt.xlabel(\"Prediction from Model\")\n    plt.ylabel(\"Loss Value\")\n\nplt.legend()\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Joint Optimization Loss\n\nThis paper proposes a joint optimization framework of learning DNN parameters and estimating true labels. The framework can correct labels during training by alternating update of network parameters and labels\n\nhttps://arxiv.org/pdf/1803.11364.pdf\n \n\n*THIS PAPER HAS A SECOND COMPONENT OF UPDATING LABELS REFER TO THE PAPER OR GITHUB HERE*\n\nhttps://github.com/DaikiTanaka-UT/JointOptimization"},{"metadata":{"trusted":true},"cell_type":"code","source":"def joint_optimization_loss(y_true, y_pred):\n    \"\"\"\n    2018 - Joint optimization framework for learning with noisy labels - https://arxiv.org/pdf/1803.11364.pdf\n    \"\"\"\n    y_pred_avg = K.mean(y_pred, axis=0)\n    p = np.ones(10, dtype=np.float32) / 10.\n    l_p = - K.sum(K.log(y_pred_avg) * p)\n    l_e = K.categorical_crossentropy(y_pred, y_pred)\n    return K.categorical_crossentropy(y_true, y_pred) + 1.2 * l_p + 0.8 * l_e","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"inputs = [x/100 for x in range(0, 100, 1)]\noutputs = []\n\nfor i in inputs:\n    outputs.append(joint_optimization_loss(tf.convert_to_tensor([1.0]), tf.convert_to_tensor([i])).numpy())\n    \nplt.figure(figsize=(20,10))\nplt.plot(inputs, outputs)\nplt.title(\"Response of Joint optimization Loss for Target = 1\")\nplt.xlabel(\"Prediction from Model\")\nplt.ylabel(\"Loss Value\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Bi-Tempered logistic loss\n\nBi-Tempered logistic loss is a generalized softmax cross-entropy loss function with bounded loss value per sample and a heavy-tail softmax probability function.\n\nhttps://ai.googleblog.com/2019/08/bi-tempered-logistic-loss-for-training.html"},{"metadata":{"trusted":true},"cell_type":"code","source":"import functools\n\ndef for_loop(num_iters, body, initial_args):\n  \"\"\"Runs a simple for-loop with given body and initial_args.\n  Args:\n    num_iters: Maximum number of iterations.\n    body: Body of the for-loop.\n    initial_args: Args to the body for the first iteration.\n  Returns:\n    Output of the final iteration.\n  \"\"\"\n  for i in range(num_iters):\n    if i == 0:\n      outputs = body(*initial_args)\n    else:\n      outputs = body(*outputs)\n  return outputs\n\ndef exp_t(u, t):\n  \"\"\"Compute exp_t for `u`.\"\"\"\n\n  def _internal_exp_t(u, t):\n    return tf.nn.relu(1.0 + (1.0 - t) * u)**(1.0 / (1.0 - t))\n\n  return tf.cond(\n      tf.equal(t, 1.0), lambda: tf.exp(u),\n      functools.partial(_internal_exp_t, u, t))\n\ndef log_t(u, t):\n  \"\"\"\n  Compute log_t for `u`.\n  \n  https://github.com/google/bi-tempered-loss\n  \"\"\"\n\n  def _internal_log_t(u, t):\n    return (u**(1.0 - t) - 1.0) / (1.0 - t)\n\n  return tf.cond(\n      pred=tf.equal(t, 1.0), true_fn=lambda: tf.math.log(u),\n      false_fn=functools.partial(_internal_log_t, u, t))\n\ndef compute_normalization(activations, t, num_iters=5):\n  \"\"\"Returns the normalization value for each example.\n  Args:\n    activations: A multi-dimensional tensor with last dimension `num_classes`.\n    t: Temperature 2 (< 1.0 for finite support, > 1.0 for tail heaviness).\n    num_iters: Number of iterations to run the method.\n  Return: A tensor of same rank as activation with the last dimension being 1.\n  \"\"\"\n  return tf.cond(\n      pred=tf.less(t, 1.0),\n      true_fn=functools.partial(compute_normalization_binary_search, activations, t,\n                        num_iters),\n      false_fn=functools.partial(compute_normalization_fixed_point, activations, t,\n                        num_iters))\n\ndef compute_normalization_binary_search(activations, t, num_iters=10):\n  \"\"\"Returns the normalization value for each example (t < 1.0).\n  Args:\n    activations: A multi-dimensional tensor with last dimension `num_classes`.\n    t: Temperature 2 (< 1.0 for finite support).\n    num_iters: Number of iterations to run the method.\n  Return: A tensor of same rank as activation with the last dimension being 1.\n  \"\"\"\n  mu = tf.reduce_max(input_tensor=activations, axis=-1, keepdims=True)\n  normalized_activations = activations - mu\n  shape_activations = tf.shape(input=activations)\n  effective_dim = tf.cast(\n      tf.reduce_sum(\n          input_tensor=tf.cast(\n              tf.greater(normalized_activations, -1.0 / (1.0 - t)), tf.int32),\n          axis=-1,\n          keepdims=True), tf.float32)\n  shape_partition = tf.concat([shape_activations[:-1], [1]], 0)\n  lower = tf.zeros(shape_partition)\n  upper = -log_t(1.0 / effective_dim, t) * tf.ones(shape_partition)\n\n  def iter_body(i, lower, upper):\n    logt_partition = (upper + lower)/2.0\n    sum_probs = tf.reduce_sum(input_tensor=exp_t(\n        normalized_activations - logt_partition, t), axis=-1, keepdims=True)\n    update = tf.cast(tf.less(sum_probs, 1.0), tf.float32)\n    lower = tf.reshape(lower * update + (1.0 - update) * logt_partition,\n                       shape_partition)\n    upper = tf.reshape(upper * (1.0 - update) + update * logt_partition,\n                       shape_partition)\n    return [i + 1, lower, upper]\n\n  _, lower, upper = for_loop(num_iters, iter_body, [0, lower, upper])\n  logt_partition = (upper + lower)/2.0\n  return logt_partition + mu\n\ndef compute_normalization_fixed_point(activations, t, num_iters=5):\n  \"\"\"Returns the normalization value for each example (t > 1.0).\n  Args:\n    activations: A multi-dimensional tensor with last dimension `num_classes`.\n    t: Temperature 2 (> 1.0 for tail heaviness).\n    num_iters: Number of iterations to run the method.\n  Return: A tensor of same rank as activation with the last dimension being 1.\n  \"\"\"\n\n  mu = tf.reduce_max(input_tensor=activations, axis=-1, keepdims=True)\n  normalized_activations_step_0 = activations - mu\n  shape_normalized_activations = tf.shape(input=normalized_activations_step_0)\n\n  def iter_body(i, normalized_activations):\n    logt_partition = tf.reduce_sum(\n        input_tensor=exp_t(normalized_activations, t), axis=-1, keepdims=True)\n    normalized_activations_t = tf.reshape(\n        normalized_activations_step_0 * tf.pow(logt_partition, 1.0 - t),\n        shape_normalized_activations)\n    return [i + 1, normalized_activations_t]\n\n  _, normalized_activations_t = for_loop(num_iters, iter_body,\n                                         [0, normalized_activations_step_0])\n  logt_partition = tf.reduce_sum(\n      input_tensor=exp_t(normalized_activations_t, t), axis=-1, keepdims=True)\n  return -log_t(1.0 / logt_partition, t) + mu\n    \ndef _internal_bi_tempered_logistic_loss(activations, labels, t1, t2):\n    \"\"\"\n    \n    https://github.com/google/bi-tempered-loss\n    \n    Computes the Bi-Tempered logistic loss.\n        Args:\n            activations: A multi-dimensional tensor with last dimension `num_classes`.\n            labels: batch_size\n            t1: Temperature 1 (< 1.0 for boundedness).\n            t2: Temperature 2 (> 1.0 for tail heaviness).\n        Returns:\n            A loss tensor for robust loss.\n    \"\"\"\n    if t2 == 1.0:\n        normalization_constants = tf.math.log(\n            tf.reduce_sum(input_tensor=tf.exp(activations), axis=-1, keepdims=True))\n        if t1 == 1.0:\n              return normalization_constants + tf.reduce_sum(\n                  input_tensor=tf.multiply(labels, tf.math.log(labels + 1e-10) - activations), axis=-1)\n        else:\n            shifted_activations = tf.exp(activations - normalization_constants)\n            one_minus_t1 = (1.0 - t1)\n            one_minus_t2 = 1.0\n    else:\n        one_minus_t1 = (1.0 - t1)\n        one_minus_t2 = (1.0 - t2)\n        normalization_constants = compute_normalization(\n            activations, t2, num_iters=5)\n        shifted_activations = tf.nn.relu(1.0 + one_minus_t2 *\n                                     (activations - normalization_constants))\n\n    if t1 == 1.0:\n        return tf.reduce_sum(\n            input_tensor=tf.multiply(\n                tf.math.log(labels + 1e-10) -\n                tf.math.log(tf.pow(shifted_activations, 1.0 / one_minus_t2)), labels),\n            axis=-1)\n    else:\n        beta = 1.0 + one_minus_t1\n        logt_probs = (tf.pow(shifted_activations, one_minus_t1 / one_minus_t2) -\n                      1.0) / one_minus_t1\n        return tf.reduce_sum(\n            input_tensor=tf.multiply(log_t(labels, t1) - logt_probs, labels) - 1.0 / beta *\n            (tf.pow(labels, beta) -\n             tf.pow(shifted_activations, beta / one_minus_t2)), axis=-1)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"inputs = [x/100 for x in range(0, 100, 1)]\n\nf, (ax1, ax2) = plt.subplots(1, 2, figsize=(20,10))\n\nfor t1 in [temp1/100 for temp1 in range(0, 100, 25)]:\n    outputs = []\n    for i in inputs:\n        outputs.append(_internal_bi_tempered_logistic_loss(tf.convert_to_tensor([1.0]), tf.convert_to_tensor([i]), t1, 1.0).numpy())\n    \n    ax1.plot(inputs, outputs, label=\"T1 = \" + str(t1) +\"  |  T2 = 1.0\")\n    ax1.set_title(\"Bi-Tempered logistic loss for Target = 1\")\n    ax1.set_xlabel(\"Prediction from Model\")\n    ax1.set_ylabel(\"Loss Value\")\n    \nfor t2 in [temp2/100 for temp2 in range(0, 500, 25)]:\n    outputs = []\n    for i in inputs:\n        outputs.append(_internal_bi_tempered_logistic_loss(tf.convert_to_tensor([1.0]), tf.convert_to_tensor([i]), 0.2, t2).numpy())\n    \n    ax2.plot(inputs, outputs, label=\"T1 = 0.2  |  T2 = \" + str(t2))\n    ax2.set_title(\"Bi-Tempered logistic loss for Target = 1\")\n    ax2.set_xlabel(\"Prediction from Model\")\n    ax2.set_ylabel(\"Loss Value\")\n\nax1.legend()\nax2.legend()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}