{"cells":[{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"markdown","source":"## Why is my leaderboard score so low, my accuracy / F1 is so much better ?"},{"metadata":{"_uuid":"072aa24f2ba5cc99c0b0fa0da9874e999067d8a0"},"cell_type":"markdown","source":"This seems to be a (beginner?) question here in the kernels/discussion and I had the same issue.\nTurns out it is quite simple:\n\n - your model sucks and you use the wrong metric! ;-) (applies to me too) \n - The evaluation metric for this competition (and therefore the Leaderboard) is **Macro**-F1\n - Some (most?) libraries default to **Micro**-F1 and simple accuracy scores behave similarily\n - Micro-F1 gives you much better score in this competition than Macro-F1, which is why your \"local\" score is better than the leaderboard\n\nUPDATE: If you are sure you are already using the correct metric and still have low LB scores, do check out this discussion:\nhttps://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/69366\nIt basically points out the fact, that the **order of the image-ids in the submission file has to be identical with the sample_submission.csv**. So reorder your results accordingly!"},{"metadata":{"_uuid":"7a810329a65cac369a2b56f67e578aa39ca66791"},"cell_type":"markdown","source":"## So what's the difference?"},{"metadata":{"_uuid":"9c7ebc4d30bb290a46e1c3600eb414dc0873e56b"},"cell_type":"markdown","source":"The difference is mainly in when/where the averages are taken, and that makes a huge difference:\n- Micro-F1 basically adds up all the metrics (true positives, ...) accross classes and calculates f1 on the averages. Most accuracy scores do the same.\n- Macro-F1 first aggregates the classes in themselves (columns) before calculating the scores for each, then calculates the average\n"},{"metadata":{"_uuid":"3ce184d5f4dc4848b64074d4560302846059fb87"},"cell_type":"markdown","source":"## Show me some code!"},{"metadata":{"trusted":true,"_uuid":"bf9f044339e789196f494db27ebe2dd9f43854a4"},"cell_type":"code","source":"import numpy as np\nfrom sklearn import metrics ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a4b1b6df97af2b0f4a3d6b4728e5359c58e3ef54"},"cell_type":"code","source":"n = 8 # number of 'training examples'\n\n# create some dummy data \ny_pred = np.zeros(n*10).reshape((n, 10))\ny_true = np.zeros(n*10).reshape((n, 10))\ny_pred[:4] = [1,0,0,0,0,0,0,0,0,0] # (play with it to see the effects!)\ny_true[:] = [1,1,0,0,0,0,0,0,0,0] # (play with it to see the effects!)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"54aa3f671283a2c87ab8fa72754cfab352080fa1"},"cell_type":"code","source":"print('Micro F1:',metrics.f1_score(y_true, y_pred, average='micro'))\nprint('Macro F1:',metrics.f1_score(y_true, y_pred, average='macro')) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d94ebef60a7f03a26d517b9abeac0172592499ee"},"cell_type":"code","source":"# Let's recreate the functions and have a closer look:\n\ndef f1_micro(y_true, y_preds, thresh=0.5, eps=1e-20):\n    preds_bin = y_preds > thresh # binary representation from probabilities (not relevant)\n    truepos = preds_bin * y_true\n    \n    p = truepos.sum() / (preds_bin.sum() + eps) # take sums and calculate precision on scalars\n    r = truepos.sum() / (y_true.sum() + eps) # take sums and calculate recall on scalars\n    \n    f1 = 2*p*r / (p+r+eps) # we calculate f1 on scalars\n    return f1\n\ndef f1_macro(y_true, y_preds, thresh=0.5, eps=1e-20):\n    preds_bin = y_preds > thresh # binary representation from probabilities (not relevant)\n    truepos = preds_bin * y_true\n\n    p = truepos.sum(axis=0) / (preds_bin.sum(axis=0) + eps) # sum along axis=0 (classes)\n                                                            # and calculate precision array\n    r = truepos.sum(axis=0) / (y_true.sum(axis=0) + eps)    # sum along axis=0 (classes) \n                                                            #  and calculate recall array\n\n    f1 = 2*p*r / (p+r+eps) # we calculate f1 on arrays\n    return np.mean(f1) # we take the average of the individual f1 scores at the very end!\n\nprint('Micro F1 (sklearn):',metrics.f1_score(y_true, y_pred, average='micro'))\nprint('Micro F1 (own)    :',f1_micro(y_true, y_pred))\nprint('Macro F1 (sklearn):',metrics.f1_score(y_true, y_pred, average='macro')) \nprint('Macro F1 (own)    :',f1_macro(y_true, y_pred))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7ff3fe02adfcf0990a6ef2dc525c22380bd27a59"},"cell_type":"markdown","source":"Obviously, those functions can be combined into one , use `axis=None` to generate micro, calculate the mean always. They were separated for educational purposes only.\n\nOf course, this doesn't help you get a better score, but it should help you iterate faster, when your score actually reflects the leaderboard better. ;-)\nGood Luck!"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}