{"cells":[{"metadata":{"_uuid":"0c993ee2aecbbd209d09581097fa84dcb78a3421"},"cell_type":"markdown","source":"You might notice, that binary crossentropy loss is not performing very well. Let's study, why's that and look for other possibilities."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"from keras.losses import binary_crossentropy, categorical_crossentropy\nimport keras.backend as K\nimport numpy as np\nfrom prettytable import PrettyTable\nfrom prettytable import ALL\nfrom sklearn.metrics import f1_score\nfrom matplotlib import pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cbf82dac9c92d0a75f9a52b11845486cc4bb12bb"},"cell_type":"markdown","source":"Let's define the ground truth with 2 possible labels. First label is in 20 % of cases, second label in 80 % of cases. We make 5 observations to make it simple."},{"metadata":{"trusted":true,"_uuid":"e1d692b461fb33db38a14e8e2c172203feabd9db","scrolled":false},"cell_type":"code","source":"# ground truth\nY = np.zeros((5,2))\n# first label is assigned to 20 % of observations\nY[0,0] = 1\n# second label is assigned to 80 % of observations\nY[0:4,1] = 1\n\n# ground truth with shape (BATCH_SIZE, NO_OF_LABELS)\nprint(Y)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8dab3170805d94ec6d4f30d427b99f8cece93232"},"cell_type":"markdown","source":"Let's calculate all the possible predictions for the first and second label. For first, you can have 0 or 1 true positive and 0, 1, 2, 3 or 4 false positives.\nFor second, you can have 0, 1, 2, 3 or 4 true positives and 0 or 1 false positives."},{"metadata":{"trusted":true,"_uuid":"64f02f01734e066175c268141404a926b598d29f"},"cell_type":"code","source":"first = {}\n\nfor TP in range(2): # TP can be 0..1\n    for FP in reversed(range(5)): # FP can be 0..4\n        idx = TP*5+(4-FP)\n        name = 'TP' + str(TP) + 'FP' + str(FP)\n        Yhat1 = np.zeros(5)\n        Yhat1[0:TP] = 1\n        Yhat1[5-FP:] = 1\n        first.update({name: Yhat1})\n\nsecond = {}\n\nfor TP in range(5): # TP can be 0..4\n    for FP in reversed(range(2)): # FP can be 0..1\n        idx = TP*5+(4-FP)\n        name = 'TP' + str(TP) + 'FP' + str(FP)\n        Yhat2 = np.zeros(5)\n        Yhat2[0:TP] = 1\n        Yhat2[5-FP:] = 1\n        second.update({name: Yhat2})","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c79d7597b028045121683aca198421c3dd5b8bcb"},"cell_type":"markdown","source":"# Binary crossentropy\nThis is the standard logloss.\nLet's calculate binary crossentropy loss for all the possibilities and compare them with macro F1-score."},{"metadata":{"trusted":true,"_uuid":"997639e0ea44ae10dbc5b26bd657414b06955fee"},"cell_type":"code","source":"t = PrettyTable(['1st/2nd']+list(first.keys()))\npltX = []\npltY = []\n\nfor name2,data2 in second.items():\n\n    row = [name2]\n    for name1,data1 in first.items():\n        data = np.stack((data1, data2), axis=1)\n        loss = np.mean(K.eval(binary_crossentropy(K.variable(Y), K.variable(data))))\n        f1 = f1_score(Y, data, average='macro')\n        pltX.append(loss)\n        pltY.append(f1)\n        row.append('{:2f}\\n{:2f}'.format(loss, f1))\n    t.add_row(row)\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ae67afa8a8318ecd38f9e4b8224bbb0c65dae3c9"},"cell_type":"code","source":"print('Displaying result')\nprint('Columns = prediction for first label (20 % of 1s in ground truth)')\nprint('Rows = prediction for second label (80 % of 1s in ground truth)')\nprint('Cell = 1st number binary_crossentropy loss, 2nd number macro F1-score')\nprint('')\nt.hrules = ALL\nprint(t)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5b9950ce8e46f5f3b9a92262e714b4989d4961fc"},"cell_type":"code","source":"plt.scatter(pltX, pltY)\nplt.ylabel('Macro F1-score')\nplt.xlabel('Binary crossentropy loss')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3a850a32a978ae48186034952cde163d1f94634d"},"cell_type":"markdown","source":"**This does not look good at all!** To maximize our metric we need a loss function, that is aligned with the metric. Here we can see many examples, when loss differs while the metric stays same (see first 5 columns). And also example when loss is same and a metric differs (see the last column of first row vs pre-last columns of the second row).\nThis misalignment between a loss function and a metric can lead to the suboptimal convergence."},{"metadata":{"_uuid":"3ee413bd859215c37b1b7f1e3b218a03d3b30fbb"},"cell_type":"markdown","source":"# Crosscategorical entropy\nFor multilabel problem, crosscategorical entropy is not recommended as well. From keras documenation: \n\n> when using the categorical_crossentropy loss, your targets should be in categorical format (e.g. if you have 10 classes, the target for each sample should be a 10-dimensional vector that is all-zeros except for a 1 at the index corresponding to the class of the sample). \n\nUsing crosscategorical entropy is therefore not optimal from theoretical point of view. In practice, however, it may still work. Let's have a look."},{"metadata":{"trusted":true,"_uuid":"a83595f03c3e41aa8843e5b374636064ab500729"},"cell_type":"code","source":"t = PrettyTable(['2nd\\1st']+list(first.keys()))\npltX = []\npltY = []\n\nfor name2,data2 in second.items():\n\n    row = [name2]\n    for name1,data1 in first.items():\n        data = np.stack((data1, data2), axis=1)\n        # nan -> 0 this is dubious, is that correct? \n        loss = np.mean(np.nan_to_num(K.eval(categorical_crossentropy(K.variable(Y), K.variable(data)))))\n        f1 = f1_score(Y, data, average='macro')\n        pltX.append(loss)\n        pltY.append(f1)\n        row.append('{:2f}\\n{:2f}'.format(loss, f1))\n    t.add_row(row)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e59736b13fc5ba51a07bf91072f10c7d4ef350ff"},"cell_type":"code","source":"print('Displaying result')\nprint('Columns = prediction for first label (20 % of 1s in ground truth)')\nprint('Rows = prediction for second label (80 % of 1s in ground truth)')\nprint('Cell = 1st number categorical_crossentropy loss, 2nd number macro F1-score')\nprint('')\nt.hrules = ALL\nprint(t)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ed27be03a64acc1db2f377fdf8c003bfbdfd8097"},"cell_type":"code","source":"plt.scatter(pltX, pltY)\nplt.ylabel('Macro F1-score')\nplt.xlabel('Categorical crossentropy loss')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1a9ac0db4131054312d49283cfdd552474fda713"},"cell_type":"markdown","source":"Now the results are not any better."},{"metadata":{"_uuid":"966c140172725e95e83c048f968d0701454841b0"},"cell_type":"markdown","source":"# Optimal loss function - macro F1 score"},{"metadata":{"_uuid":"34bb29d17e4848b86dfc4f15bea9e43dd1f927fd"},"cell_type":"markdown","source":"The best loss function would be, of course the metric itself. Then the misalignment disappears.\nThe macro F1-score has one big trouble. It's non-differentiable. Which means we cannot use it as a loss function.\n\nBut we can modify it to be differentiable. Instead of accepting 0/1 integer predictions, let's accept probabilities instead. Thus if the ground truth is 1 and the model prediction is 0.4, we calculate it as 0.4 true positive and 0.6 false negative. If the ground truth is 0 and the model prediction is 0.4, we calculate it as 0.6 true negative and 0.4 false positive.\n\nAlso, we minimize 1-F1 (because minimizing $1-f(x)$ is same as maximizing $f(x)$)\n\nI took the function in [this great Kernel](https://www.kaggle.com/guglielmocamporese/macro-f1-score-keras) and took the liberty to modify it:"},{"metadata":{"trusted":true,"_uuid":"f73bb97ae2a8a776bbcfdb559f37d61da74eaca6"},"cell_type":"code","source":"import tensorflow as tf\n\ndef f1(y_true, y_pred):\n    y_pred = K.round(y_pred)\n    tp = K.sum(K.cast(y_true*y_pred, 'float'), axis=0)\n    tn = K.sum(K.cast((1-y_true)*(1-y_pred), 'float'), axis=0)\n    fp = K.sum(K.cast((1-y_true)*y_pred, 'float'), axis=0)\n    fn = K.sum(K.cast(y_true*(1-y_pred), 'float'), axis=0)\n\n    p = tp / (tp + fp + K.epsilon())\n    r = tp / (tp + fn + K.epsilon())\n\n    f1 = 2*p*r / (p+r+K.epsilon())\n    f1 = tf.where(tf.is_nan(f1), tf.zeros_like(f1), f1)\n    return K.mean(f1)\n\ndef f1_loss(y_true, y_pred):\n    \n    tp = K.sum(K.cast(y_true*y_pred, 'float'), axis=0)\n    tn = K.sum(K.cast((1-y_true)*(1-y_pred), 'float'), axis=0)\n    fp = K.sum(K.cast((1-y_true)*y_pred, 'float'), axis=0)\n    fn = K.sum(K.cast(y_true*(1-y_pred), 'float'), axis=0)\n\n    p = tp / (tp + fp + K.epsilon())\n    r = tp / (tp + fn + K.epsilon())\n\n    f1 = 2*p*r / (p+r+K.epsilon())\n    f1 = tf.where(tf.is_nan(f1), tf.zeros_like(f1), f1)\n    return 1 - K.mean(f1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"96cfa513ccba8ae1d970d06222258dc74c69af33"},"cell_type":"code","source":"from keras.layers import Dense\nfrom keras.models import Sequential\n\nmodel = Sequential()\nmodel.add(Dense(2, input_dim=2, activation='relu'))\nmodel.add(Dense(10, activation='relu'))\nmodel.add(Dense(2, activation='sigmoid'))\n\nmodel.compile(optimizer='adam', loss=f1_loss, metrics=['accuracy', f1])\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a283aa85c6c1b3b279a091b7af0d679bbac25de7"},"cell_type":"code","source":"t = PrettyTable(['1st/2nd']+list(first.keys()))\npltX = []\npltY = []\n\nfor name2,data2 in second.items():\n\n    row = [name2]\n    for name1,data1 in first.items():\n        data = np.stack((data1, data2), axis=1)\n        loss = K.eval(f1_loss(Y, data))\n        f1 = f1_score(Y, data, average='macro')\n        pltX.append(loss)\n        pltY.append(f1)\n        row.append('{:2f}\\n{:2f}'.format(loss, f1))\n    t.add_row(row)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8d4479a0c752a436c88d24eca344fe16a2cdbd02"},"cell_type":"code","source":"print('Displaying result')\nprint('Columns = prediction for first label (20 % of 1s in ground truth)')\nprint('Rows = prediction for second label (80 % of 1s in ground truth)')\nprint('Cell = 1st number focal loss, 2nd number macro F1-score')\nprint('')\nt.hrules = ALL\nprint(t)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"998a1c4b6f01041a63f93d625996ec456afa7e26"},"cell_type":"code","source":"plt.scatter(pltX, pltY)\nplt.ylabel('Macro F1-score')\nplt.xlabel('Differentiable F1 loss')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"efa7a100ef7ac1fd6ed9fb5c9f291c492660776c"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}