{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.\n\n","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true,"_uuid":"a9709906bba5ac1e57e534af4f08f6b22ba9c5bf"},"cell_type":"code","source":"%%javascript\nMathJax.Hub.Config({\n    TeX: { equationNumbers: { autoNumber: \"AMS\" } }\n});","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Introduction - 2nd Edition\nThe original purpose of this notebook is to understand [Matthew Correlation Coefficient (MCC)](https://en.wikipedia.org/wiki/Matthews_correlation_coefficient), especially in the context of VSB competition (If you are a VSB competitor wanting to visit the old note book, please see Section Introduction -- 1st Edition, below).\n\nHowever, after the competition ended, I seemed to be able to understand MCC in a more general sense. I was able to derive the MCC formula into an intuitive form which will benefit all other future usages. Therefore, I decide to revise this notebook, and hence here's 2nd Edition part."},{"metadata":{},"cell_type":"markdown","source":"## Demythifying MCC\n\n![basic definitions](https://upload.wikimedia.org/wikipedia/commons/thumb/2/26/Precisionrecall.svg/330px-Precisionrecall.svg.png)\n\n### Goal : to show that MCC can be reformulated as\n\\begin{equation*}\n\\text{MCC} = (Pos\\_Precision + Neg\\_Precision - 1) * PosNegRatio\n\\end{equation*}\n\nwhere $Pos\\_Precision = \\frac{\\mathit{TP}}{\\mathit{TP}+\\mathit{FP}}$, $Neg\\_Precision = \\frac{\\mathit{TN}}{\\mathit{TN}+\\mathit{FN}}$, standard definitions of [precisions](https://en.m.wikipedia.org/wiki/F1_score) for positive prediction and negative prediction, respectively (see picture above). The $PosNegRatio$ is defined as \n\n$$PosNegRatio = \\sqrt{\\frac{(PosPredictionCount)(NegPredictionCount)}{(PosLabelCount)(NegLabelCount)}}$$\n\nwhere $PosPredictionCount = (TP+FP)$ means **total number of positive predictions** in the test set, and the other three variables can be interpreted in the same sense.\n\n## Interpretation : \nAbove, you can see that MCC score consists of 3 main variables, which we would like to maximize simultaneously.\nNote that, before seeing the simplified MCC version, we may have intuitively thought that **by increasing both positive precision and negative precision, MCC score should increase**, and perhaps the score is an average of the two precisions. **This intutiion is heuristically acceptable in many cases**, but not all, as explained in the next paragraph.\n\nThis intuition does not work especially in the VSB situation where we have such a imbalance distribution of positive (4%) and negative examples (96%). If a good MCC were just somehow be acheived by good precisions, we may do this by guess only one (very confident) positive example, and get 100% positive precision. Even though, we may predict all the rest as negative, due to the highly ratio of negative example, we still get around 96% of negative precision. In this case, we will have very high two precisions, but still a bad MCC score. The explanation lies in the third term, $PosNegRatio$.\n\nHere, $PosNegRatio$ plays an important role. This ratio forces us to make the number of positive and negative predictions **roughly the same as the ground-truth number of labeled examples**. So in the above case, we cheated by predict only 1 positive example, where there are around 4% of 11592 (public test data) $\\approx 464$, so we shall get a very bad score in the factor of $\\frac{1}{\\sqrt{464}}$.\n\nTo conclude, the correct intuition is to maximize the two precisions, and also $PosNegRatio$ together!\n\n## Deriving the intuitive formula\nWe shall start by its standard definition.\n\n\\begin{equation*}\n\\text{MCC} = \\frac{ \\mathit{TP} \\times \\mathit{TN} - \\mathit{FP} \\times \\mathit{FN} } {\\sqrt{ (\\mathit{TP} + \\mathit{FP}) ( \\mathit{TP} + \\mathit{FN} ) ( \\mathit{TN} + \\mathit{FP} ) ( \\mathit{TN} + \\mathit{FN} ) } }\n\\end{equation*}\n"},{"metadata":{},"cell_type":"markdown","source":"First note that $(TP+FN)(TN+FP)$ is constant with respect to the given test set; they are in fact $(PosLabelCount)(NegLabelCount)$. Therefore, to simplify the formula, we define \n\n$$\\kappa = (PosLabelCount)(NegLabelCount)$$.\n\nWe also define \n\n$$ \\alpha = (PosPredictionCount)(NegPredictionCount) = (TP+FP)(TN+FN)$$\n\nHence, now\n\n\\begin{equation*}\n\\text{MCC} = \\frac{1} { \\sqrt{\\kappa\\alpha } } \\times (\\mathit{TP} \\times \\mathit{TN} - \\mathit{FP} \\times \\mathit{FN})\n\\end{equation*}\n\nNext, we will focus on the second term of [RHS](https://en.m.wikipedia.org/wiki/Sides_of_an_equation). Observe that by definitions,\n\n$$FP \\times FN = (PosPredictionCount - TP)\\times(NegPredictionCount - TN) $$\n\nIf we plug this into the MCC equation, we will have $(TP \\times TN) - (FP \\times FN)$ equal to:\n\n\\begin{equation*}\n(TP \\times NegPredictionCount) + (TN \\times PosPredictionCount) - \\alpha\n\\end{equation*}\n\nThen, by the defition of **precision** (see the picture above), the first term is\n\n\\begin{equation*}\nTP \\times NegPredictionCount = Pos\\_Precision \\times \\overbrace{PosPredictionCount \\times NegPredictionCount}^{\\alpha}\n\\end{equation*}\n\nSimilarly,\n\\begin{equation*}\nTN \\times PosPredictionCount = Neg\\_Precision \\times \\alpha\n\\end{equation*}\n\nTherefore, we finally have :\n\n\\begin{equation*}\n(TP \\times TN) - (FP \\times FN) = \\alpha \\times (Pos\\_Precision + Neg\\_Precision - 1)\n\\end{equation*}\n\nAnd hence, \n\n\\begin{equation*}\n\\text{MCC} = \\sqrt{\\frac{\\alpha}{\\kappa}} \\times (Pos\\_Precision + Neg\\_Precision - 1)\n\\end{equation*}\n\nBy defining $PosNegRatio = \\sqrt{\\frac{\\alpha}{\\kappa}}$, we finish our proof."},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"acd0dc42d598b33ad4c52a12b64e3eb5f82aaf40"},"cell_type":"markdown","source":"# Introduction - 1st Edition\n\nGreeting everyone. I think almost everyone here agrees that this *VSB Power Line Fault Detection* is not an easy competition. The performance of the best public kernel is so difficult to beat and improve. \n\nIn my opinion, one of the main reason is that the MCC formula \n\n\\begin{equation*}\n\\text{MCC} = \\frac{ \\mathit{TP} \\times \\mathit{TN} - \\mathit{FP} \\times \\mathit{FN} } {\\sqrt{ (\\mathit{TP} + \\mathit{FP}) ( \\mathit{TP} + \\mathit{FN} ) ( \\mathit{TN} + \\mathit{FP} ) ( \\mathit{TN} + \\mathit{FN} ) } }\n\\end{equation*}\n\nis quite complex and not so intuitive compared to usual metrics like **accuracy, F1**, etc. \n\nEventhough this section on wikipedia https://en.wikipedia.org/wiki/Matthews_correlation_coefficient#Advantages_of_MCC_over_accuracy_and_F1_score gives us some explanation why MCC is nice :\nunlike the  **'accuracy' metric**, MCC adjust score well in the class imbalance situations, and unlike **F1** which concentrates only on positive-class prediction, MCC also force our performance to do well on both postive and negative prediction.\n\nHowever, due to its complexity, we will have a hard time to gain insight from our current MCC score, e.g. if we score 0.694, \n\n- how many positive/negative examples that we are correctly classified? \n\n- Does our algorithm have a good precision?\n\n- If we are able to correct one more positive / negative example, how much will MCC increase?\n\n- To get 0.750, how many more data must I classify correctly?\n\nEven though we may not answer the above questions perfectly, in this problem, we may have a way to approximate them.\n\n## Goal of the kernel\nTo demythify some aspects of MCC, and understand impact of 'one more correct' positive and negative example to your current MCC score, so that you can better design a stretegic improvement on performance. Note that this is not a programming based kernel. But there is a small code in the end for illustration on MCC calculation."},{"metadata":{"_uuid":"848f546d936794931bfd02f0c744972f8d72bcb0"},"cell_type":"markdown","source":"# Where shall we start?\n\nTo achieve our goal on MCC understanding, please note that if are able to guess the four values, *TP, FP, TN and FN*, we will understand the model performance on the test data much better, e.g.\n\n## How many positive/negative examples that we are correctly classified? \n\n**Answer** For positive-class and negative-class accuracy (or **recall**) calculate $\\frac{TP}{TP+FN}$, and $\\frac{TN}{TN+FP}$ respectively.\n\n## Does our algorithm have a good precision?\n\n**Answer** For positive-class and negative-class **precision** calculate $\\frac{TP}{TP+FP}$, and $\\frac{TN}{TN+FN}$ respectively.\n\n## If we are able to correct one more positive / negative example, how much will MCC increase?\n\n**Answer** For one more correctly classified positive example, calculate the following formula to see how much your score change, and you will know the impact of *one-more correct example*:\n\\begin{equation*}\n\\text{MCC} = \\frac{ \\mathit{(TP+1)} \\times \\mathit{TN} - \\mathit{FP} \\times \\mathit{(FN-1)} } {\\sqrt{ (\\mathit{TP+1} + \\mathit{FP}) ( \\mathit{TP} + \\mathit{FN} ) ( \\mathit{TN} + \\mathit{FP} ) ( \\mathit{TN} + \\mathit{FN-1} ) } }\n\\end{equation*}\n\nIn the case that we group the three phase as one example, and always predict the thress phases together, the formula will be:\n\n\\begin{equation*}\n\\text{MCC} = \\frac{ \\mathit{(TP+3)} \\times \\mathit{TN} - \\mathit{FP} \\times \\mathit{(FN-3)} } {\\sqrt{ (\\mathit{TP+3} + \\mathit{FP}) ( \\mathit{TP} + \\mathit{FN} ) ( \\mathit{TN} + \\mathit{FP} ) ( \\mathit{TN} + \\mathit{FN-3} ) } }\n\\end{equation*}\n\nAnd do similar calculation for one more correctly classified negative example.\n\nIf we understand more on this metric, we may able to devise the new strategy, for example,\n\n*if you know that your model is quite precise on positive examples, but still not cover enough of them (low recall), and by MCC simulation, correctly classify one more positive will increase more score than correctly classify one more negative,  you may try to train your model to 'cover' more positive examples [e.g. by re-balancing the class]  *\n\n### So our goal is to make a hueristic just to **guess** these four values *TP, FP, TN and FN* !"},{"metadata":{"trusted":true,"_uuid":"f171f1b41ec971905664dc1d7f120e3b17337d44"},"cell_type":"markdown","source":"## How can we calculate these four values?\nAccording to basic algebra, if we could get four equations, we can solve for all four variables (regarding public test data which is 57% of the total test data). And actually, we know of two trivially :\n\n$$TP + TN + FP + FN = \\text{Number of your test data} = 20337 * 0.57 = 11592 $$, and\n\n$$TP + FP = \\text{Number of your predicted positive examples} =673*0.57 \\approx 384 $$\n\nNote that in the second equation, I put the predicted number of positive examples of my models which is 673 (you can check your number easily at the summation of 1 in your submission.csv)\n\nFurther, note also that other similar relations such as $$ TN + FN = \\text{Number of your predicted negative examples}  $$ cannot be used, since it can be derived from the above two equations. In other words, we need two more *linear independent* equations.\n\nAnd where we need some hacks. Firstly, according to the post by Putalay and Sergey ( https://www.kaggle.com/c/vsb-power-line-fault-detection/discussion/82868 ), we can roughly probe **the ratio of the positive test data**, i.e. \n\n$$\\frac{TP+FN}{TP+FP+FN+TN } = \\frac{TP+FN}{11592}$$\n\nHere, according to Putalay's method, I test with the test data number 17511 and get the score 0.06. That's mean this example is indeed positive (faulty signal), and its implies that the ratio of positive data is around 2.3%. So our third equations is : \n\n$$\\frac{TP+FN}{11592} = 0.023 \\Rightarrow TP+FN = 0.023*11592 \\approx 267  $$\n\nSo far so good, now we have 3 linerly independent equations, we need just one more equation in order to decode for the four values. In fact, the fourth equation is the MCC equation itself where you can put the MCC you get from submission : \n\n\\begin{equation*} \\label{MCC}\n\\frac{ \\mathit{TP} \\times \\mathit{TN} - \\mathit{FP} \\times \\mathit{FN} } {\\sqrt{ (\\mathit{TP} + \\mathit{FP}) ( \\mathit{TP} + \\mathit{FN} ) ( \\mathit{TN} + \\mathit{FP} ) ( \\mathit{TN} + \\mathit{FN} ) } } = 0.694\n\\end{equation*}\n\nTheoretically, it's done. We can use all 4 linearly independent equations to solve for all the values of *TP, FP, TN and FN*. However, since the last equation (MCC) is non-linear, so it can be difficult in practice to solve it together with the first 3 equations. To simplify our process, we have to make some assumptions which may not 100% correct, but is sensible. Our assumption is based on the fact that we have abundant negative examples, and our model has no problem to accurately predict the negative ones at all. Hence, most of the time, our model will have $FN \\approx 0$ (which I found out to be quite correct, see verification code below). By using this assumption together with all information we have from the 3 equations we get:\n\n\\begin{equation*} \n\\frac{ \\mathit{TP} \\times \\mathit{(11592-384)} - \\mathit{FP} \\times \\mathit{0} } {\\sqrt{ (384) ( 267 ) ( 11592-384 ) ( 11592-267 ) } } = 0.694\n\\end{equation*}\n\nHence, we can now solve for *TP*, and can then in turn solve for other values!\n"},{"metadata":{"trusted":true,"_uuid":"e5c94725794c49bad4b7c8a83170a462812c31d7"},"cell_type":"code","source":"N = 11592\n\nTP = 0.694*np.sqrt((384)*(267)*(N-384)*(N-267))/(N-384)\nTP = int(TP)\nprint(TP)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d91c01e50ace65ac650ae70c58d3a86c1631a199"},"cell_type":"markdown","source":"Great!! We have *TP* around 223! And by using this number, we can better approximate *FN* (instead of 0) by using the fact that $TP+FN = 267$. In turn, we also have a better guess on *TN* as well."},{"metadata":{"trusted":true,"_uuid":"21c3b67d72cc60ce4082711e7f9348274f6ae638"},"cell_type":"code","source":"FP = int(384- TP)\nFN = 267-223\nTN = int(N-TP-FN-FP)\n\nprint(FP,TN,FN)\nprint('sanity check : total number of examples = TP+FP+TN+FN = ',TP+FP+FN+TN)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"58cc5748b44ccbf6cefb8a58ed0f2713ea9d6a49"},"cell_type":"markdown","source":"Before continue, note that the $FN \\approx 0$ assumption can be verified easily by writing the following kears callback metrics :"},{"metadata":{"trusted":true,"_uuid":"e210ae9b4fce0e0c40797129ec2f1ae894a38b3c"},"cell_type":"code","source":"def neg_precision(y_true, y_pred):\n\n    y_pred_pos = K.round(K.clip(y_pred, 0, 1))\n    y_pred_neg = 1 - y_pred_pos\n\n    y_pos = K.round(K.clip(y_true, 0, 1))\n    y_neg = 1 - y_pos\n\n    tn = K.sum(y_neg * y_pred_neg)\n    fn = K.sum(y_pos * y_pred_neg)\n    numerator = tn\n    denominator = (tn + fn)\n\n    return numerator / (denominator + 1e-15)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"74f4290736b68868b14697645de8d24752298bc0"},"cell_type":"markdown","source":"and let it print out together with MCC metric when compiling your keras model:\n```python\nmodel.compile(loss='binary_crossentropy', optimizer='adam', metrics=[matthews_correlation,neg_precision])\n```\nand, provide that our model fits the data well enough, we should see this metric almost always print out around 0.95++."},{"metadata":{"_uuid":"fdee9bef6cabfa3005090364bd38b187f6d2fdbf"},"cell_type":"markdown","source":"# And now it's done\nNow we can answer the question we posted in the beginnning easily. First of all, let us check whether our four numbers will produce MCC score similar to the original one (0.694)"},{"metadata":{"trusted":true,"_uuid":"92c24d7ae9ba2345e3c99bcdaa9bb2f05364065d"},"cell_type":"code","source":"tp,fp,tn,fn = TP,FP,TN,FN\n\ndef mcc(tp,fp,tn,fn):\n    return (tp*tn - fp*fn)/((tp+fp)*(tp+fn)*(fn+tn)*(fp+tn))**0.5\n\nprint(mcc(tp,fp,tn,fn))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7cfe3d62c2aaa0146ca4b5a4e23df7a8965c43b0"},"cell_type":"markdown","source":"We get 0.688! I think this is good enough and so the four number that we get provide quite accurate guess!!\n\nNow, if you want to know how much score will be increase/decrease if you change your prediction by one data point:"},{"metadata":{"trusted":true,"_uuid":"f2d10e9c8401188ccbebb80f727c44649e3ba787"},"cell_type":"code","source":"#correctly from 0 to 1\nprint(mcc(tp+1,fp,tn,fn-1))\n\n#correctly from 1 to 0\nprint(mcc(tp,fp-1,tn+1,fn))\n\n#incorrectly from 0 to 1\nprint(mcc(tp,fp+1,tn-1,fn))\n\n#incorrectly from 1 to 0\nprint(mcc(tp-1,fp,tn,fn+1))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"10fad4187324e95f2671ee35be9ff05494c6253d"},"cell_type":"markdown","source":"So now you can see the MCC effect of classify one more correct/incorrect test data! Here, correctly guess one more positive and negative data will give you around 0.0022 and 0.0009, respectively. Note that if you group all the 3 phases as one guess, this effect has to be multiplied by 3."},{"metadata":{"trusted":true,"_uuid":"1af0f44e01719890347a7766cb2ba8869ff91d57"},"cell_type":"code","source":"#correctly from 0 to 1\nprint(mcc(tp+1,fp,tn,fn-1) - mcc(tp,fp,tn,fn)) \n\n#correctly from 1 to 0\nprint(mcc(tp,fp-1,tn+1,fn) - mcc(tp,fp,tn,fn))\n\n#incorrectly from 0 to 1\nprint(mcc(tp,fp+1,tn-1,fn) - mcc(tp,fp,tn,fn))\n\n#incorrectly from 1 to 0\nprint(mcc(tp-1,fp,tn,fn+1) - mcc(tp,fp,tn,fn))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c2a43300ede093ef84d0b5a70acc0c3c67753b28"},"cell_type":"markdown","source":" ### To get 0.750, how many more data must I classify correctly?\n We can answer this question by just play around our mcc calculation"},{"metadata":{"trusted":true,"_uuid":"19b2ae48245249b71f174484ed7b4e0e308c95bf"},"cell_type":"code","source":"print(mcc(tp+28,fp,tn,fn-28))\nprint(mcc(tp,fp-60,tn+60,fn))\nprint(mcc(tp+14,fp-30,tn+30,fn-14))","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"markdown","source":"So you need to correct more 28 positive data, or around 9 if you group 3 phases togehter. Alternatively, you can correct more 60 negative data or around 20 for 3-phase answering. Just 20 more correct, from ten thousand examples, sound easy, right? I find it very difficult ;)\n\nOkay. That's all for now and hope that you can find this kernel helpful in some way ;) "},{"metadata":{"trusted":true,"_uuid":"60496ea5f1966f1e57d6b1e83d5b50b90130a96c"},"cell_type":"code","source":"print(mcc(tp+6,fp-12,tn+12,fn-6))\nprint(mcc(tp+12,fp-12,tn+12,fn-12))","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}