{"cells":[{"metadata":{},"cell_type":"markdown","source":"# This is a one stop notebook for the problem metric - **AUC**"},{"metadata":{},"cell_type":"markdown","source":"# As always, remember to upvote if you liked the content :)"},{"metadata":{},"cell_type":"markdown","source":"# This Notebook provides a very intuitive understanding of the ROC-AUC Metric. Stick till the end of the Notebook, I promise it will be very fruitful."},{"metadata":{},"cell_type":"markdown","source":"# Basic Terms\n\n**First let's get some basic terms out of the way before proceeding further.**"},{"metadata":{},"cell_type":"markdown","source":"# Confusion Matrix\n\nI am very sure that everyone is familiar with confusion matrix and terms revolving around it. But bear with me.\n\n![](https://glassboxmedicine.files.wordpress.com/2019/02/confusion-matrix.png)\n\nThe ones we are interested are: **Recall** and **Specificity**"},{"metadata":{},"cell_type":"markdown","source":"# Recall aka TPR (True Positive Rate)\n\n![](https://miro.medium.com/max/444/1*HgxNKuUwXk9JHYBCt_KZNw.png)"},{"metadata":{},"cell_type":"markdown","source":"# Specificity\n\n![](https://miro.medium.com/max/308/1*f7NmMcQtfes1ng7jtjNtHQ.png)"},{"metadata":{},"cell_type":"markdown","source":"# FPR (False Positive Rate)\n\n![](https://miro.medium.com/max/306/1*3GhDfiuhvINF5-9eL8g6Pw.png)"},{"metadata":{},"cell_type":"markdown","source":"Its no secret that we want **TPR** to **Increase** and **FPR** to **Decrease**."},{"metadata":{},"cell_type":"markdown","source":"# ROC Curve\n\n\nThe **receiver operating characteristic** curve is plot which shows the performance of a binary classifier as function of its cut-off threshold. It essentially shows the true positive rate (TPR) against the false positive rate (FPR) for various threshold values. Let’s explain more.\n\n\nMany of the classification models are probabilistic, i.e. they predict the probability of a sample being a cat. They then compare that output probability with some cut-off threshold and if it is larger than the threshold they predict its label as cat, otherwise as non-cat. \n\nAs an example your model may predict the below probabilities for 4 sample images: \n\n    [0.45, 0.6, 0.7, 0.3]\n\nThen depending on the threshold values below, you will get different labels:\n\n    cut-off=0.5:                    predicted-labels = [0,1,1,0]\n\n    cut-off=0.2:                    predicted-labels = [1,1,1,1]\n\n    cut-off=0.8:                    predicted-labels = [0,0,0,0]\n\nAs you can see by varying the threshold values, we will get completely different labels. And as you can imagine each of these scenarios would result in a different precision and recall (as well as TPR, FPR) rates.\nROC curve essentially finds out the TPR and FPR for various threshold values and plots TPR against the FPR. \n\nA sample ROC curve is shown in below Figure.\n"},{"metadata":{},"cell_type":"markdown","source":"![image.png](attachment:image.png)","attachments":{"image.png":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAQgAAACwCAYAAADzGPBTAAAgAElEQVR4Ae1dB3gVVdrWVX/72p913V1XXcvq6u66u7ZV144iKFhRFAWUItKkSa/SpRNaIITee6+hJKSRkJAESEglCSE9hPQp7/+8BydeUm9yJ/fOTc55nsncTDnlnTnv+eacr1wFmSQCEgGJQDUIXFXNcXlYIiARkAhAEoR8CSQCEoFqEXAJQZSWluLEiRM4duwYAgIC5CYxkO+Ak94BPz8/JCcnV0sIFU84RBDs4NOmTUN8fPwV+QYGBmL27NkIDQ294rjxT2JiIrp06YK5c+diwYIFmD9/vtwkBjW+A54LFsDTU26OYLDYaxGGDB4s+qbRF2vbO0QQlALY0Xfs2FFeTkJCAkaOHImVK1eKfWpqavk54wevmTNnjvGv3EsEJAJOQuD06dPw9va2uzSHCIKlLFy4EDt37iwvcNeuXfDw8BD/jxs3DpQmjKTrOgoKChAUFIQZM2YYh+VeIlAjAml5xZi5LxojN0VgzNYoudUVg53RGLv7LCbtOI1u0zdgnqdXjXjbnnSYIDw9PbFnz57yPLdt2yZERR6YOHGimGcwTqqqKsikZ8+eGD16tHFY7iUCNSIQFJ+FZpMPYvLOKCzxjceiw7FysxeDgGQsPHAGK7uPwM7Pu2L0tOWY7720RrxtTzpEECUlJRg/frz4nODEY1lZmZh8HDVqlJh/GDJkCGJiYmzLg6IoiIqKwqxZs644Lv+RCBgIFJYoSMwsQFzGJaTkFGJNYCI+neOLjPxi4xK5txeB+BigRyfg7ZeBDStwKiQY3suX2Xu3Y8uchw8fFnMQgwYNgq+vL8LCwlBUVIQ1a9Zg8ODB2LRpEyg1VEyc1OQkpkwSgaoQmLU3Gp/P9UWPZcFia7fgGPqvDkVeUVlVl8tjVSFQVAQUFEDbswPqjz2hn4oSV0VERTpvDoJSAzdKEtxTOjBScXH1bB8bGysJwgBK7ish0GdVCMZtj0LMhYuISM4VW3peETiHJVPtCOghwVB6doW6fStAzGxwO3nypPMIovaqVn2FJIiqcWlsR5OyCrDSPwHeR+PF3AHnD2ralvnFw+twLFpM9cGawKTGBkfDt6esDOr82VDeeAHq8IHQU1MqlSkJohIk8oCrEFjml4A3Jx3EoLVhGLo+HIPXhdW8rQ/DwLVhGL4xHBEpua6qttuVqysKtIR46DnZUOfNgrZtU7VtkARRLTTyhLMQKC5TUVSqYvqeMxi8PhylqgZN16FqWq2bomnOqmajKEc/lwTlh++gtG8DPSuz1jZJgqgVInlBQyJw6MwF9Fp+HL2Wh6Dl1EOYuvt0QxbXpPPWAnyhvP8WlC5fQTseYBcWkiDsgkle1FAIzNoXjXbzj2HXyfM4EHVBLFM2VFlNNV8tyB9a5Elo4WHQNm+AzhULO5MkCDuBkpc5joCiaoi+cBHHYjNwLDYTxxOy0WtlCEZsOul45jKHSgjoublQvRdCeetFqIsXVDpvzwFJEPagJK8xBYH4zEv4ct4xfL3AX+grfL8kWPxeFZBoSv4yk18R0EtKoAzuD+XdV8UkpF5Y+OvJOvySBFEHsOSljiFwMjkXn831RWBcFnIKSpB+sRhZl0qhaFJfwTFkf71bP58KdcdW6Ckp0A77QI+J/vVkPX5JgqgHaE3lllJFw/YTKZi8Iwo/7zyFKbvqv83Ycxp9Voag9czDcp6hIV6g0jJou7dD+fAdKF07CoIwoxhJEGag2EjzyCooxaezfdFz+XFwMnHqrtMObSSZdUFJKCj5VYO2kULn9GZpx4OgtG4OdcoE6NnZppUvCcI0KN0nI+ocnMsuREJmQbVbck4R/GOz0HLaYQTHm/fCuQ9K1q8pVyPU5YuhbVwD5OZCj4s1vdKSIEyH1PoZrg5IQrv5/ujsFYTOi6veungHgUZP707xwanUPOs3qonVUAsPhfJdByjvvAJty7oGa70kiAaD1roZD994EmO2RCA67aJQUaaacsWNE4onU3KRkJGPMkVqK1rlaerQAdpQjB8OtV836LHmSw22bZUEYYtGI/xdVKrgaEwG1gefw8aQZGwITsbHs4+Cdg8yuRcC+pGDUAf1hRYfe3meoQrXCGa3SBKE2YhaLL+wpBx8Mvsoui09jh/XhKHvqhPotSIEAXG16+FbrClNtjr6hQtQx426rPA0YQyoAOWsJAnCWUg7oRya8VOnQP1lo3rBgVNp+NrTHxn5JZdrYGPr74QqySIcQUBToRcVQgsOgtK9C7Tdv/pydSTbutwrCaIuaFn42viMS/hpayR6rQgVUgIlhf5rwvDJbF98NscX+cXSu5KFH1+lqmnx8VB7d4Ey+SeguBh6aWmla5xxQBKEM1B2Qhl7ItPQesZheB6KxbJjCVjqd3nz9o2Hz6kLQqpwQjVkESYgoG1eC6XF61A6tYMW6GdCjvXPQhJE/bEz5c68wjKcSr2Ik8l5iEip33b6/EVM33cGXZf8GjLAlMrJTJyKgJ6UJOYX1GVeUBfOAVwkNdg2WhKELRou+O11JA7vTjmErzz98fVCf7Hn77psXy8MQKsZRzB6S4SUFFzwDB0uMjcH6tSJUD5tDT30uMPZmZmBJAgz0axHXqM2R2Dg2hNIzi4UGzUc67MlZxfhovTiXI8n4NpbaFylfNMWSpv3oe3dBZT8Mpns2mqVl+50gmCkrHPnzomYGOW1AJCeng6G3dOqcCHWmJ3WcmJxyi7pRcn2XWgKv/WUZGgnw4HUFGhLFoL/WzE5lSDo7n7mzJno27evCJ5juCVnUN+xY8eK2Jxbt26thFNSUlJ5eL5KJ938AAli0o5Tbt4KWX27EaAW5Ia1UN59BerEMUAVA6LdeTnhQqcSBKN3M8Ymg/EywpYRqJcRuxmzkwF0pk+fXt5sEkhOTg4YcMf2ePkFjeCHJIhG8BDr0ATVewGUZv+DOnWCXU5j65B1g1zqVIJg0F5G8Wb0LIbSY7RvJkoQPXr0QKdOnUR0LaOlvG737t344YcfGm1sTkkQxtNuvHt6c1J3bIYeexZ6cAD0k2Fu01inEoSPjw+8vLxEVK0pU6YgLi5OAMWgvYzgferUKYwYMaLSPMTZs2cbbWStCTuiMHG7/MRwmx5Tl4oy/gSjVnVuB+X9t8XvutxuhWudShAZGRkYM2aM+Lzgp0ZERISYnPT29saECRNA0pg/f36lkGkkksYQm7OoTBXu1vZEpGFfZBr2RaWhw0J/TJRzEFboC6bXQU+Ih9KxLdRBvaEnuKdxnFMJgk+AAXv5mcGJR3b8CxcuoLCwENu3b8fGjRuRlZVV6UE1llWM6Av5wgEL/SxQqamTdyDaLfDHzvDKIc8qgSAPuA0C2u4dUBd5ApkZ0E5HATYxaN2mEb9U1FSCyM/PF9G6zQahsRBESGIOPp/rhzNp+SLydG5hqbCRoHGVTO6PgJAYBvWD8uqzUD1mujUxGE/DFIKg7kL//v1xzz33YN68eejXrx9IFmalxkIQoYk56LgoAJmXrKUMY9Zzaqr5CHrXdGhLF0H54kNox3wbDRSmEAQ/GR5//HG8+eaboB5D8+bNTV2WbCwEEZGchw4LA5CWV9xoXqCm3hA9KhLK0H7QfPYB+ReBAvMGRitgawpBcOKxQ4cOghSOHj2K3r17Y8CAAaa1rzEQBAPSMrTcF/OO4bwkCNPeDVdlpFPhaYUXlLf/B6XTV9Cjo1xVlQYt1xSCSExMxFNPPYWbbroJ9913Hx588EGcOXPGtIq7O0FQBF14KBbNfj6IvqtDpc2EaW+GazLSL12CnnEB6sCe0FZ4A8WN95PRFILgYwoMDBRLmNRjoL4DVybMSu5OEGWqhgFrTogYlMk55uFiFr4yH/sQYLwJdfJYKKOHQudqW2njJQYDEYcJIjk5WWg5fv/990J1moZY7733ntBrMApxdN8YCGLI+nDhONZRLOT9rkFABKZp+yGUj96FdnAf4ASHsa5p6ZWlOkwQnTt3xtVXX42rrrpKrGLccccduPbaa4Vew5VF1f8/dycILmMOXR+Olf4ySG393wLX3KmnnYdOfYb9e6F6zAAyMlxTEReV6jBBpKSkYNKkSRg4cCAoTcTHxyPXZK+7jYEgBq0Lw/Jj7qlN56J307XFlpSIqFVKy9egbmq4wDSubWTtpTtMECyCVpceHh4YPHgwhg4dij59+oh5iNqLt+8KqxJEZEqucPbC+YVBa8MwsIqNxNB/dShaTDmEzSFSY9K+J+7aq/T8fCjD+kN56yWo82ebGuvStS2re+mmEAStNG+99VZcd911YhXjmmuuwbp15rGuVQlibXASPph5REgGqwMSscq/6o2fFltCU5CRL/Uf6v6KOvGOixehBfhDz8iAumoZ9PDL1sZOrIHlijKFILhyMXLkSGFQ5efnh59//lloVJrVWlcTRGGpgoSMS4hJy8fZC5e3uPRLmLgjSqxOmNVOmY+LENA06KHBUL79EkqXjk1aYqj4BEwhiLVr1+Kxxx4TilLNmjXDc889h2HDhlUsq97/u5og1gYlosXUQ+jgGSA0IakNye2DmUcx9+DZerdL3mgNBLQjB6C8/RIUWl0yQrYMLlT+YEwhCOZGU23qQnTs2BGtWrUCJy/NSq4miOl7o/HDihAkZRUKSYJBauIyLiEhowD5xYpZzZT5OBkB7eBe6CHB0BNioR095OTS3aM4hwmiqKgI27ZtE+bbbHJkZCSWLl0qfD2YBYErCOLkuVzM3heNWXuj8eHMIxi3NdKs5sh8XIyAHn0Gap/uwi+kttf54exc3Pw6Fe8QQZSVleH9998XOhB333033nrrLfH7//7v/7B48eI6VaSmi11BENP3nMFrEw5g2IZwjN4cgWNnm9b6d03Pw63PFRZC+aEblA6fQwsKcOumOKPyDhEEdR6uv/56rF+/HpMnTxarGFu2bAFdxJWaGBXIGQRRpqjCNwNjWHJScsyWCDBmhUyNAwG6flM9pkOPjxPzDPQTKVPtCDhEEDTIevjhh0UpJIXXXnut9hLrcUVDEwT9MzB4TdclQei94rjYWk0/jAU+sfWorbzFSgjoeXlQveZDafYy1EH9gOxsK1XP8nVxiCBICnQSM23aNPz444944IEHsGjRIuFX8vRp84LBNDRBcOmS8wxLfONx6Ew69kel4Uh0OtIvSr0Fy7/BtVRQ8/OF0qEt1HWrgDI5oVwLXJVOO0QQNMyi7QXtMLhRUcr4PWfOnEqF1feAMwiii3cQGL5OJvdHQM/MhDrpJ6grl0EvKAD/l6l+CDhEEIqiiNULzkUwGI6xsUO7k8s5ShCdFwchLr2gfijKu6yBgK5D27kdykctoHz+ATS/I9aolxvXwiGCcFa7nSFBSIJw1tNsmHJ0untjgJqfhkGdMAbIkXMNZiDtdIJgFC3GvoiOjr6i/r6+vuJ4SEjIFcf5jySISpDIAwYCShm0FUuhdP4K+tkzTcKJi9F0Z+xNIwgGwaElJ+NoBgcHV1l3alf+9NNPgghor8FI30ysxNSpU4UFKMmgYjp//jzMnNOomH9iZoH8xKgIihv8rycmQOnyJZTmr4oI2Y3NYawVHoEpBBEVFSVWMOg4hh356aefrtLce//+/ViyZAk4d8GVD4baY2LUrCFDhoA2HSQDIzE258GDB8U5OsY1K+UXlSE2PR/RafmIz7yE3RHn0W6+P2LlHIRZEDdoPrS21M+nQo86CXXyT9DPmLdi1qAVd8PMTSEIWnJ+/fXXQgpgBG9KEvRsXTFVDN7Lwpl4//Dhw8EQfCQOEggT422QfBj5m/E7zUpzDsSgzRw/ITV09Q4SwWx6Lj+OC9LbtFkQN0w+ug792FEo7T6BOnd2owhM0zBAmZerKQSxadMm4dW6RYsWIhL33/72NyxbtqxSLRl2j58SnH8YN26cMOiis5kVK1Zgw4YN4jjJhfYdtsnsT4wBa0+AQXO5ehGVkie2CxeLIeNb2aJuvd/qPA8orz0PdcJI6BfSrFfBRlgjUwiCuNCa86GHHsK9996LHj16gJ8HFRPVr+fOnStiZtChDKUN6lIwHieD+dKvxI4dOyreZtokJVWod59MxcezjmBtUFKlcuQBCyKgKNAjwgQhaFs3Qdu13YKVbLxVMoUguDLBuYfjx4+jpKRmV+A8zyjfJBBKCsb1ly5dQmY1Ci1mrWKEJGaj5dRD6LfqBE6l5jXep9pIWkbfDMqA3lA+bg4tIryRtMq9mmEKQQQFBYlgOZQg6PLey8sL2SbqvJtFEEejM9B92XFkSBVqy7+l2tloKJ+1gtKxLfTjQdKJi4uemCkEYdR98+bN+NOf/iTUrbkyYVYyiyB8YzLRe0UIsi6VmlU1mY/JCDDwrbZnpwhMox0+CEaxksl1CJhCEHQSQ78Qf/jDH/DEE0/gm2++gRWNtQLjstFrRQgy8mv+DHLd42i6JevZWVA9pl2ehJw5tekCYbGWm0IQBw4cwKeffordu3cLF/hmt7GuEkRRqYLQpBwcjcmA7y9bQFwWZu+PRufFwciUBGH2I3I4P3XaJCgfNoe2b6dcvnQYTfMycIgg0tLShH4CY3FS76Ffv37o3r07GG2LGpVmpboSxOEz6fjMwxedvALRbUkQvlsShG5LgvHFPD9M2H4KhaWVV1jMqqvMx34E9NRkqLOmQouKgB4bAz35nP03yyudgoBDBEHrTTqp3b59O55//nmhQfmvf/0LzzzzDLZu3WpaA+pKEKsDkoST2dTcQmRfKkFmfrGISUHJoViSg2nPpd4ZlZZA3bAOyvvNoHz9KbTTUfXOSt7YsAg4RBBG1ajPwM8LI1HxadeuXca/Du/rShBrg85h+EbpLs5h4BsoA/30KSgfvwv153GgxyeZrIuAwwRB1ehHH31UKEh99NFH5U5saYxlVqoPQQzdcBJlqmZWFWQ+jiJQVipcv6kzp0FPT4eeme5ojvJ+JyDgMEHMmzdPqFnTNyXtMdq2bStidJoZwFcShBPehAYsghKD2q09lHdfhbpuNaBK128NCLepWTtEEDSqogYkTb1pS3Hx4kXk5eWJvSu9WvMTQ0oQpr4n9cusuBgoKYG2cT2UH3tDi4upXz7yLpch4BBB0Ks1PVnTMOv2228XLvBvvvlmMC6Gp6enaY2qqwTBKNqSIEyDv14ZMSCN8k1bUNkJtKeowjanXhnLm5yKgEMEQYlhz549whflxo0bhT8HBsyhZ2uucJiV7CUIWoZGJOeCptu0t1A1aZ9p1jOwNx89LxfK8IFQXn8B2tTxQiPS3nvlddZDwCGCMJpTWFiI0aNHIzw8XEgU77zzjvjUMM47ureXIBj05vulwXh3ig+W+cU7Wqy8vw4I6KWl0FNToWdmQBk3CtphnzrcLS+1KgKmEAQnKkkKjOj973//G4zwPWHCBNPabC9BZBeUiqjbh0/LGXLTwLcjI0arUr77Bmqv74AiGUvEDsjc5hJTCGL8+PHo0qULXnnlFfGZQaLo37+/aSDUhSA6LQ5CQGyWaWXLjGpGQPPZD+XDt6F0agct8BjdgNV8gzzrVgiYQhA01vrtb3+LRx55BLTovP/++4VvCLOQqCtB+MXIQClmYV9dPlpoCPTkZGhHD0NbuRQovtILWHX3yePuhYApBMEm09nLiRMnhIfqpCRzvTVJgrDOS6VnZUKdMg7Km/+FusM8dXrrtFDWxBYBUwiCzmUZm5PLm9dccw169eolHM7aFuTIb3sJgkF4v/UKwLFYKUE4gnd194oViu++hfJZa2g++wATI7hXV6Y87loETCEIOq297777sGbNGtDXJJ3G0L29WclegkjJKUT7hf4IO5djVtEyHwCMP6Ht3AY9JQWazwHoKckSlyaCgCkEQQ/VrVu3Lofs22+/RZ8+fcr/d/SHvQQRnXYRX3v6Iz5DeiFyFHNxf3ExtNXLobzzKpTe3YCLF03JVmbiPgiYQhB0Z3/nnXeiVatW+OCDD3DLLbfA39/fNBTsJQi/s5lglG7pEMYc6LWD+6C897rw2aDnSKnMHFTdKxdTCIJNpuNaup174403qnRdb8BCM3BGyaoYg5OermfOnImzZ88al5bv7SWIjcHn0HdVKEoV6RCmHLw6/tAZAHfRPGi+PhARrM5K+4k6QtioLneYIGiPQR2IAQMGID29ZgUlrm4wNufq1auFJyqqahuJcxf/+9//EBAQYBwq3ycmJsLDw6P8/+p+zPc5ixEbL0frqu4aebx6BPQTIVA6toHS7GXoRw5Vf6E802QQcIggGHz3ySefFL4g7rrrLrz44os1+qTct28fli5dKkLrTZ8+XViBEumYmBgsX75cSBZcKjUSbStYBqUTBtapLU3aeQpTdss4jbXhVOk8TVZKiqEM7At1cD9oCVJNvRJGTfSAQwTBz4HbbrtNhNDjb1p0MlJWdYmfF/Q2xaA5/JygghUTY3NOnjwZXbt2xapVq8pJhtfxHvq7pK1HbWnYxnAsPFI5Onht9zXl89reXVBHDb4cDJfh7KTVZVN+HSq13SGCYIzNBx98EPHx8SLILgnCz89PGGoZEbNsS2Q070mTJoGRuLjywU+OsrIyYRFKC1DG9uSnBCUHI9HnBAP4zpo1yzhU5b5M0dBnZSg2HpdLcFUCVOGgdi4RyuC+UN54ESqtLvN//dyrcKn8twkj4BBBkBiuuuoq/OY3vxEbf1999dVCYWrBggWVYKVCFSUEBujdv3+/0LxkHkaimnZVk5S8prZAPOkXi4QlJ93cy1Q9AnpxMfSCAnCFQu3V+bL9RPWXyzNNHAGHCILzA1SIYkBebpQCaNlJKYCjfnWJUgMTCYNbbcmeVYzTqXlCi5IRu2WqGgH9VCSUTl9BXTj/cuwJO7CvOid5tKkg4BBBOAskewji2NlMdFjoL1zcO6teblMOiXjtKihv/BdK987Qz8iJXLd5di6uaKMhiJ3hqejsHYhLxdIhqu07paemQM/Ngbp4IdSVS6XrN1tw5O9aETCVIBiPszZdiFprVMUF9kgQq/wT0WNZMEqU2j9Zqiii0R2ikpM6YxKUdq2hx0YDv877Nrq2ygY1HAKmEARdzrVp0wY33HADqN9A9/fUjDQr2UMQHvujMWR92BUrIGaV72756DGnoXz6HpQ274POY+k0ViaJQH0QMIUg6MGaofcYhm/v3r0ikO/YsWPrU58q77GHIMZsicT0PWeqvL+pHNTjYqCfPQM9IQ7qMi/o6ReaStNlOxsIAVMIgroNjO5N3QYuX5IouJRpVqqNIBhB64eVoVjpn2hWke6VD+0nVnhDeetlqPNnu1fdZW0tjYApBMF5hxdeeEG4naNfiCeeeEIoQZnV8toIIr9YQZclQdh9MtWsIt0qH9Vj5mUPTwvnQb8ol3nd6uFZvLKmEATbSJdzVI6iCnVKSoqpza6NIBi9u/PiQHCps6kkPScL2vo1EKsU4Segx0Q3labLdjoRAVMIgp8Vf//73/Hf//5XbJyPoJcps1JtBJGUVYBOXoGITG0CkaIVFZrfUShffgDly4+hxVY2jzcLd5mPRMAUgqDRVb9+/YRfSpp+06s1I26ZlWojiJCEbPRYdhwpOY3fs7IedgLKB82hjhkOLVXanZj1jsl8qkbAFIKomHWHDh3AWBlmpdoIYntYCgauDUNBSeNdztP27IK6bhWQmQGqTEOGFTTr9ZL51ICAKQQRGBgovEm1bNlSOH2h0VZVxlo11KPGU7URhOehWIzectl0vMaM3PAkfTOoA39A2ZsvQvWqbADnhk2SVXYjBEwhCJpvf/LJJ0JBig5rGcDXMMgyA4vaCGLCjlOY2ggdxeiaBtVjOpS2raTVpRkvksyjzgiYQhA04SZBNFSqjSAGrgvDgsONZ7JODwmC2q87tNBg6NnZ0AsLGgpama9EoEYETCEIZnLvvfeKgL0HDhwQE5TJyeZNoNVEEKWKJvxArAk0N5pXjag10Ek6bVFnTYXy9stQh/QTS5gNVJTMViJgFwIOEURRURF8fX1FwJzrr79eOI/h/AM3+oUwK9VEENSB+GZRIPZHua9aMe2oKCXonG/o1wuaDGln1qsj83EQAYcIgt6fbrzxRgQHByM1NVVslBzohZrOZMxKNRFEzIWL+HL+MYSdyzWrOKfmo6enQ50wGsrowaC3J5SUOLV8WZhEoCYEHCIIuoK7+eabhTcp+qI8cuSI2A4fPozz58/XVG6dztVEEIFxWfhsji/OZZlHSHWqnAMXa4f2Q2nxGpQ2raD5HpIW2Q5gKW9tGAQcIgg6nb3pppuu+LQwPjHsiWNhb5NqIoh9kWki3J47RdPS085Dz8uDumk91MljodObtEwSAQsi4BBBxMXFCR8QjGkRGhoqomUxYhY/OWibYVaqiSDWBSfhu6VByC++7OfSrDIbJJ+SEqhLvaB8/B60Qz4NUoTMVCJgJgIOEURCQoKI5M0O3JCpJoKgktSgtWGg23srJ7p9Uwb0hvLOK9BWeAMmztFYud2ybu6NgEMEQWUozkNUFQOjOlhyc3PBeBrFnJD7JTEEH0P4Vbc0WhNBTNwRhRl7z1jWpZqelQHteCD4WaGtWQk9umk7tTGeudy7BwIOEURdm8hl0WnTpqFnz54i1J7h8t7Hx0dEzho0aFCVUcEZrauqOQ3qQPy45gRWB1jQUYyiXLa6/LwVlO87AEWFdYVLXi8RcDkCTiUI2mwwAE5aWpoI3svlUKZLly6J/Y4dOzBlypRyUBhhi3MZVL4isVRMeUVl6LYk2JI6EOraZVBe/jfUyePkJGTFByf/dxsEnEoQO3fuxMqVK0XwXobSsw3US72JCRMmXGEmztic9DUxcOBAERW8IqoX8orRaXEguNRpiVRaBm3XdujRp6GfDIN29LBlP30sgZeshOURcCpBHDp0SETf4vwDJQVjcpNEQAOvZcuWVQkYV0uqCr2XkHFJaFGePu/6uJL62WiovbpCefMlaD77q2yHPCgRcDcEnEoQdIX/008/4eeffxbu8RmeLysrCx3aEOEAABQ0SURBVBs3bgRNxRnGj7E1KqbqCOJ4fDa+8w5CqqsdxTAwTffOUDq2gRYWypiCFZsg/5cIuCUCTiUIIkQC2L59e7laNgkiIiJCHNu9e7dYzaiIZHWrGFtPpKDfqlCXOYrRAvyges4Dg9SIcHZy6bLio5P/uzkCTieI+uBVHUEs8DmL4RvDoTtZSVnPyoTqMQNKs5egjBoqomXXp13yHomA1RFwa4IYuy0Sk3eecirGwonL1o1Q2n4IbfdOQJcx7Zz6AGRhTkXArQmi/5oTmOfjHEcxemIC1LEjoG7dyHVZ6DnZTn1QsjCJgCsQcFuCUDUNXb2DwKC9DZpUFdrm9VDefhHK159Bjwhv0OJk5hIBKyHgtgSRVVCCDp7+2BvZcJaQWsFlSUEd9iO06ZOgX5JRq6z08sq6NDwCbksQMRfy8eW8BnIUQ6vLuTOhdPkaekryZUcuDf8sZAkSAcsh4LYEERifjbZz/Ux3FKPHxkDt0QVKy9egLlkIlLqBGbnlXitZocaCgNsSxJ7I82i/KADZBea4aNNzsyHiXTKs3aRxwj9kY3nIsh0Sgfoi4LYEsTYoCd2XBzuuJMVJyAP7oLRtDXXxfLlsWd83qQ730YpXbtbFwPZRui1BLPCJxdAN4VBUB9SaVRXqlPFQXn0W6tSJ0urS9s1ogN8khezsbKFFS5+lcrMeBnQ+zWdkuGJwW4KYsD0Ks/bVL+S9XnAJ2vFg6JkZUDeshXZYun9rAD6olCUtdmnqX1paKix6FUWRe4thQOdPJAnDBYNbEgSlhv6rTmBDcN2D5einoqB8+yWUT1tDT6r7/ZXeennAbgToOYyjk0zWRoBGlXl5eaKSbkkQuYVl6OQVAJ/TdQuWo52KFC7m1Z6doEdGyPkGJ7+nDUUQtNVhsOgVK1YIF4hsFolow4YN8PLyQlhYmGgpPZrt2bNHuBbYt29fnVwl1gcqulE0M/xDTXVgR2ZbbX2s8Hp6Y2NozC1btohYNXT3SNcKa9euxYULF4Q1Na2qbZNbEoTH7NmiDaFJOei5/DjenXoIESn2BcthrEvN7+hlv5BHDsngNLZvgxN/kyBycrJRpmpIyysWZvqpuUWwa8spwvncIhSXqZVqzA7Qp08f4Yxo/PjxoN9TeiCjm8Jt27ZhxIgRiImJwfr16zFmzBgcPHhQ/OZ1RuKISatidhrWkyI2/ZTk5OSgsLBQfBoxDzpqpqczfqPTEpmJnZAkxeuNRE9oM2fOFOI6yYqdkB2PiXkyL5ZF3yjstMY5foLxWl5jb2L+I0eOhKenp9gbefH+cePGCYKYPn26sJhes2YNiBFJk21ke1hPQ2LgPW5JEHM8PARenofi0GLqIawPPlfrCoaefgHq9J+hvP4stEVz7cVbXtdACLDjFebnIS6zQJB8+4X++MYrwK6N13ZdEoTwKiKocTTk6MlOR2JgECeSgtGBly5dKryVsWMwGpyR2NGZ2JmnTp2KsWPHCp+olES2bt0q7qejIroh6Nq1K+bMmSM6IDswy5o8eTIYJGrSpEkYMmQI1q1bZ2SN48ePg17TmIKCggQxGeWzHPplpbc0lsXOzfrSaTOdKo0aNUpcb9SfeZw6dUpIRJSKSHQMLWFMJLK9JADO7cyYMQN07WgkkiTLGz58uDjOe+mhjUGu6HSaG+9lCE0juSVB8EHlFpVhxKaTGLkpwmhL9XtVhTJ8AJSPW0I7sBcokwpP1YPlnDMkiLzcHBSVqohIyUNYUg7Cztm/nUzOxcWiys+RDofee+89tG/fXnQexmhhp+AnBRPF6x9//FGMplWNzHSFaOsLlWTDjsSRmc6NmD8lD3ZAOljmSE1SIiH06tVLdHY6XJ44cWI5kPSvSmIiCZEI6E7xiy++EPnyOnZQ+kbhMZIDCYOdlPexLLbHtqP7+/sLEiORsa4kLYMg6LOVsXD5P/dHjx4V9eD/CxcuFI6YSJzMm1ILSY2kwDYw8TeJx0huRxBJCfEYNWEq+q8JR7OffbDEN95oS6W9HnsWmtcC6EkJ0CJPglKETNZAgATBTmd24icGR3F2IhIBCYIjJv/nKGx0co7omzdvFqI1vZQZBMJPDnZgfnJwZYWBoEgSHKXZ8Tdt2lTuVZ0drH///qJDUzzn6M/r+Zuz/0ZinsyDeQ4dOlR81nDk5vc/JRHWkZ8WPEYJIyUlBZxH4P+UIjp16iScNRv5sS7s/Nx4P/M3CIJEQ09tlGy459wH28FPo379+glph4TCTwkeJ2nRaZPhCJoEQfIwktsRxLnEePwwYgI+n3sMuyPSqvwORWEhtDXLobz1ApRu30I3MTaoAZzcO4ZAQxEEX35jBGQHIjGwAw4ePBi9e/eGt7e36BQkBYrv7OAUxfPzLxvfcR6Anblv377Yu3ev6Ljs1AMGDBCjNkdcfrtTHGfiKM7z7Gj8ZOF3Pv+npGAkzknwOraZrhR5np11165dQvIgGTBROiGp8VrGmOEngVHv8HD7LIdZD5LUd999J6QWkgDrzE8nlsdySWSsE6UNkh4JlHXn3APL5jkjuSVB9BkxQTiojU2/7CLfaIzYUzMv7ASUb7+C5jUfeqGMQXEFPhb5p6EIgh3C6Ly2v1keOwmPGYmjKjsAJ+jYsYzE+3mtEZWe5MHOY+hs8LxxPY/ZBotiXrzXkEiYJ68hcTG0g+1kp1E/Y0KTUgClKn76MH/ex/+ZlyEhGHWsac98eZ/RVrbTKIN5G2TIfFlX1pmJpERpwmgbj7klQfQdeZkgTqX+6sFa50yzx3Sx6bm50LMt4v6+pifZhM81FEFYFVKSii2RWLGeJCRutsktCaLXsPFo5xmAqIzLE096wDEo7T+H8sE70LZtsm2f/G1RBDhq8RueoxxHR45wjXnjY+DIbOU2GpID68hnwmdD3Q1D4nALRamkxHi83H4QmnsEIi4lR6xIqLMmQx3WH3pawzmMsWg/c9tq8QWkeMsJOU7oyc16GHCplRO7JAwmpxNEQECAmMnlTKuRKIZx3ZkTQbbKK8b5+IQEfPfjGOyZtRyFPbsB4aFASTGg/aqYYlwr99ZGgCMWRym5WRcDQ6rgm+RUgiA7cRmGs7pcljImhKjtxiUX6jpw7bhiSsnIxMLmbwAtXgE85wCl5viAqFiO/F8iIBG4EgEqhLG/2puusvfCqq7jMhCXnDh5Q0KgdhgTl2CoQ84ZXy5DGbO3FHOoJTaeSiYPPwjvLp2wZMUKeC9fLvJhXlbcCOj8+fOFQg0Jz4p1rFgnrs9ziZD7iues+D9xpcISbTDcBWPiSIy5pMp3xIq42taJWp5cDuWSsL3JIYIwgveSAKi0Qd13Jq77UsmD36QkCOP7h9dxGYYaa32pUHLyJIKCg4WGGbXMrLpxDZ5rzWwjic+q9TTqRSUcKt5Qm49KOvzfOGfVPXGlFGoMLlatp1EvKkNRP+P7778Xmp1UxTbOWXUfEhIiiIyKWfYmhwiChEA1URZMzTVqn/FblCMBSYBKHYZ2l22FOKk1f8EC20OW/03FFarTukviUheVdAxydod60zLT1v7BHerM0ZjLiO6SnPqJQYmAImHnzp2FcgYZldpcnDWlptkPP/xQZWxOKnqQgd0p0VqO0o+7JE4UcyTj55+7JL68hhTqDnXm+893nvog7pK4LG2Yy9tTZ4ckCBZAiYEAESyOWvyfiRpexqSlPRWR10gEJALWQ8BhgrC3SZyPoNRQkW05sclRw3Ypxt48G/I6iuasF5dvSX5MPEaDHI7M6enpDVl8vfKmKjEx5ihRMdFuwdbcuOJ5V/1PiZM2CsbAYtSDxzl3UvF9Mc67as+Bj5/U9Blhm7iiR+yt9rnBhQLOoVUcrPmO8D229Rth2x7jt1MIgjrltJ0fNmyYmJ8wxF6+AJxV5ecIJ9OslGgGTJ8AnHClGMnEl3XZsmXCbwBNfJ3lXcgeXGigRCs+mhrT2MhW/4Si+9tvvy0MiezJy1nXcGKS7wQx5vyDkahbwzZwBp4vuFUSBwquWLDOfG+TfnFxyAl51pfzETRL5/tulUTjsZYtW4pJa6NOVG7jhDD7HacIalIfdwpBcEZ90aJFgsXYsTg6MPE3SYIjCIG1SiKB8WFzpKAtvjHZRwnCkCa4OsMX2SqJCjBcAaBRz9y5c8snVEkcXIL79ttvwVUnKyVOZtOwiATGAYSjGTGmObdBDlaSLDkgcNCglygOFMaEKgcQEgb9MXBvJemS7/Lo0aOvMP+mnhJ1lEgMxJ2LC9UlpxAE7e/psIOJS4VcEmIig5EsWEE2wuh84qQL/1BvnQTBuvETiCsxtixLwiO5VeWsxFXVJpFRgiCG1GA1yIDY82UgvitXrrQMxsSJREa9GI5odKrCDsgXmsuzrC9HZcNZiqtwtS2Xn5usE21QuELHQY+Jk9esL83JqThoK73Z3u+q3xx8jfeBdaCeiaEsxXcmMjKy2qo5hSAortMTDoHlcqixGkAxh99BFDUJvFWSIUGQyPjpY7tuzNGCIibnVKyUOF/C0YsYU+IxPotIDlxlevHFF9GlS5dys2Ar1J0SBEczdjyOZIZJMzsZiZm+HKpaJndV3Sk58HOIREYJh+I7E5fAqTDFRF8QfJ+tlEgQtp9wdCbDgZrzKZaQIDjSkgzYsfgNR3GYHYzLLfxm5nErjRR8uHTQQQmHnY4vKl9Y1rljx4745ptvhONQK01I8VOCD5140rciyY0SECf/eI5ORAwnLFZ5efn8iS9x5gjHkYzSG98FDhgcla30XvBzhyMv31eSGCclOWnNSUDWlbhbTbLkANeiRQvhVIef8yRjvrfUX+LnEgduDojVJadIECycs6ZkVo5wtg4v+BKz0lb61mR9KaqzXpz9ZwcznHLwheBozYkpjnhWSuxcxJirFZxQtf0EothL7K2WKE2SfLlEToy55zwE3a7RK5LV3gu+CxzgKE1wBDZWhjh4kChsMbcC1pxIZb34zlLyMerLPd8Vwwy8uro6jSCqq4A8LhGQCFgXAUkQ1n02smYSAZcjIAnC5Y9AVkAiYF0EJEFY99nImkkEXI6AJAiXPwJZAYmAdRGQBGHdZ3NFzahtSuUcah7SnR91S6pSLKNNABW5Kto2XJFZNf9wJp5LusyfZXGmvi6JmpC8lysoXH0w6keNVC4J1jVxNYP6HFwCZTQtruXXtkrAe6y28lHXdlvpekkQVnoaNdSFthRXX301/vjHP+Lee+/Fq6++WiUJUAP0mmuuqbUjVVUUlX2uuuoq/PnPf8aDDz4oyiJR2Ju4BPzcc8+JpTSqS7dr107cSr0BBrepa+Iy4gMPPIAbb7wRDz/8MO6++248+uijYvm5qrxITP/4xz+EbkJV5+WxuiMgCaLumLnkjhdeeAGvvPJKedkcnbkmT3XlDz/8UIR4o64DO/m1114rztFW5LPPPhPKXVwHZ6Jex1dffSW0K401cSNThrL7zW9+U64q/Ne//hVvvPGGOE33gh988IFQrjHWzqlMxvxp50GjKhopUWFo3759ePzxx3HzzTeLYDOUehhOj0o5DJXHRKUtaiAyUduTeRvq+OIgIPRMbr/9dqFMxWOUaEhg1P4ztF3ZdtqgsGwqV/F8mzZtRPvZ1s8//1xEqLKa+rPRRqvvJUFY/Qn9Ur9mzZrhrrvuwpdffik6AD8FqOr79NNPCxsLjrSMWE09e3YqfpLccMMN6NGjhyAPBqWlifIjjzwiwtd9+umneOaZZ65Q9qJKOaUPdlzacNxxxx3CpRqD0N5yyy1CjfjZZ58VmnkkhOuuu044BWL8SZZDxRt2UEodb775ppBEqFTEgLbstAyFR8mE6aGHHhJkQw3Pxx57THRuto92JEaiItp9990nrBFpHEV7jd///vdC6YeGUv/5z3/E9ZSoGPuSx1h/SixUwPvLX/4i2t6qVSuhak6JRKa6ISAJom54uexqEgQ7AtW8qe5NfXoauVHPniMnPz04WlIV+NZbbxWmyOwYTz31lIiSTUKhjQY7MD19dejQQYzwtmb2NP3leXZKfmZQOuDI3LZtW7z++uui7fyM4DV0v/fRRx+V508pgWTAc9SapU2CIfEwH5IE7yXp0JEuy6A6ODvxSy+9JMiCnw/vv/9+Ocbs0CQUkgkJ7f777xefGCQOSgckDKo8k1hYR8673HTTTUIlnmRHSYqfNiRVEhw1CmWqGwKSIOqGl8uufv7558s7qVGJbt26iY587Ngx0Rk/+eQTYUREyYHksXr1atGR+UnBDkITdZ6jeM84k7SB4KSmkfiJwXkOdj5b61WSCYmGiZ8a7Hgsk+bNJIqPP/4Yd955p7Cb4CcKJZXu3buLEZ73kCBat24t7uccxT333AMSHj+R2Pl5P606actAi1MjkQhuu+228k8MtokEREmBEgOlJ35GcU8C4SQp60/piRIH60ni4iQnpYqqHOkYZcl91QhIgqgaF8sd5SjLiUnbRE/bnLhjtGeOyBylaWXITkUfC//85z/FyPree+8JEZvOgvmbedG6k9fbrnZwLoCdqqIRGo2oOEnI+yipkFhoN/Hkk0+KPJo3b4533nlHjNAU8dmRWY/rr79eOCRhlGpKM0w02mMnpzTDxM8ZfvaQQJ544okrjLNIEJQOWJ6Rfve73wlDI0oolC4oDVEqofREOw7mxbkTtp91ohRDcm3fvv0VbTXyk/uaEZAEUTM+ljlLo5uKbs44Ucfw75wUpGETlyk5ilKUZ+KsPucSOIqyszGxE3G5kKMqf9smTuTxXlvSMM5zeZFLjTTPNxKv5zyIkReXGGngxnw5iUqRnvmRcEhOTJQaeI1tGezMnKC0DVnPa5kH5xJIRkaicRfzY1mUgijRMD8jf05kUqph/sSH8yHcbMsz8pL72hGQBFE7RvIKiUCTRUASRJN99LLhEoHaEZAEUTtG8gqJQJNF4P8BGhNJHv7T6hkAAAAASUVORK5CYII="}}},{"metadata":{},"cell_type":"markdown","source":"As we can see from this example, the lower the cut-off threshold on positive class, the more samples predicted as positive class, i.e. higher true positive rate (recall) and also higher false positive rate (corresponding to the right side of this curve). Therefore, there is a trade-off between how high the recall could be versus how much we want to bound the error (FPR).\n\nROC curve is a popular curve to look at overall model performance and pick a good cut-off threshold for the model.\n\n•\tROC curve can help you to choose a threshold that balances sensitivity and specificity in a way that makes sense for your particular context.\n\n•\tYou can't actually see the thresholds used to generate the curve on the ROC curve itself.\n"},{"metadata":{},"cell_type":"markdown","source":"# AUC\n\nThe area under the curve (AUC), is an aggregated measure of performance of a binary classifier on all possible threshold values (and therefore it is threshold invariant).\n\nAUC calculates the area under the ROC curve, and therefore it is between 0 and 1. One way of interpreting AUC is as the probability that the model ranks a random positive example more highly than a random negative example.\n"},{"metadata":{},"cell_type":"markdown","source":"![image.png](attachment:image.png)","attachments":{"image.png":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAKoAAACQCAYAAACYl/BJAAAbx0lEQVR4Ae2dd1hUV/rH3b9+T57djWt2NdGoa++xx4ZgwRKjRtREQxKNLWrcqNhi7L2LFVSIBaSIBVEsKMzcKXcaMIAttqiIyZrEWFBEiuD397wHLo4u6Mwwg/fOXJ6H5/Y7957zue85bznvqQT5Ty4BCZRAJQk8o/yIcglABlWGQBIlIIMqiWqSH1IGVWZAEiUggyqJapIfskJAjYuLw7Zt2+TSduMSyC/MQ8aT67iVdwN7tT/i1z9+sak0KgTU8ePHo06dOjY9mHyydEsgpyAHKQ8MOPcwGbtvbcKcixMwOm0AvIz10DywGrxWt0LGw+s2vWCFgOrn54eWLVva9GDyyeIvgfzCfDx99hSpmSac/D0am68vxZi0gfA190RHrhY6q/6NQVxnjFD0x0R+OCYGfo2Za6ZCZVQiNyfXpheUQbWpuNz35CcF2bibdwf6e0rsurUJq67OxpBED3ya5Ik+2pbooqiDT5Xd4MeNhh83Cns0O7BfEw5Or8S55AuIDNmHresDkKhLQqIxCdnZ2TYVpgyqTcXlHiffz/sTV7IuIOFOLJZfnYk1P8/Bl+be8NY1RV9NK/RRtEI/ZVvMUk3AItUM7NXsQpz2BNS8Cil8KpL5FCTySUg1pUEZzyEu7hQUCQpotVrodDro9XoZVPdAyXFv+Xvuf6G9Gw/DPQ6rrn6PKee/xLDEbvDQ1EVfVWt8quiOIQovLFRNwxb1akRpwqDhNVDxHIPRzKeypZE3Qc8bwPM8+zeZTEhISMCSJUsQExODxMREBinBKoPquPpzuTtlFzzGo6cPwf15AtG3QzHv0iT4pnhjsLELuirrw0vVAF9wfTBa6YOVqnnYq9mJw9qDEABM5JNh5lNg4hMZkASljteVgCkASksB0sWLF+PgwYMMUuG4DKrLoWX/C+UW5uKXJ+lIy0zE9vTV2HBtEetLDjS1Rx9VS/RQNsFYbjBmcuOxTrUER7WHcVi7H1peW9xsJyOJT2ZQEow6Xg++DCgFAIXly5AmJSW9ALMMqv31Kvkrrz2+hLMPkxCSsRULLk3C+LTB6ME3ZkrOAGUHDFV6YYlqFlarFiBacwAJ2tPQ83qk8GklQBZJSn2ZUlIA8VVLS0gPHTqElyGla2VQJY/b61+ATEF5hblIuq9FzO1wTDs/AhPODkEvXTN4quujO9cEwxQ9MYubgB3qzaz5JilJgCTzZlB/koA08MZiKVnUn3wVfNYeo34nOXaouS8LUhnU19exJM/IzH/ATEIn/4hGUPp6jEzph8+SvdBb8wG6KuthuNIbk5S+WKmey5ScmOI+JWncRZp3MpOcJD2tBc6e84xGI+uX7tmzB/v37y9Vkgr3lSWqJFF88aEfPc3Ez1kXse+XH7HsynT4mDqhr6EV0769FS3wDTcUi7kZiNLsxQltLLS8hklJ6k+SskPSskjBKV3JEWBx5JK0+d27dyM6OpopTQbDc82/tN+RQQXwrLAA+flPX6x92npWgLzS9uNZGfv/9xbO2HM75xeY7mtw9LdIfH9xHL4094KHuh68uIYYpOiMmapvsFm1GrHaGGajpCab+pUEZJE27lxJWRpolvuoD7pz506sWrUKarWamZ0sj5e27vag7lkzE129uqF3r57w/XY+0u+Ri64Qx0I24aNePdHbuztGT1+B37KKQE46vR8D+/aAdy9vLNgaifxnzkDxxXvmFDxhSs/h2+GY/dM49DW0RCeuNj7i2mCEsj8znodognFIGwUjbyyxUxKUhleYg0oDwpn7yGhPklSAlPqnr5OkwvO4PahTB3WCx/AfcPmCCW3e/xemBJ7EuVPBqPFubWw9qESK4TQ6130PXy8Nx+3LGjR456/4bnUIzEl6HIw+gmJ+XyTLAVs/Z11C2K3tWHx5Csg81Fv7AbopGmGM0gcLVH6I0oYxr04Sbwb9k9JDfUwyCZVlpxQq/U0sCVKSpD/++COTpLZASs/r9qDO/sIbPXxnw6hLgHf71th1woCVYz9G5+E/lOAWtmgU2vYegcClk1GreT9klhxx3MqDvLu4+CgNuzI2YuKZoeiiqQMPRT0MUHTA99xEhGl2QcHHw8SbWDNOUJLEfBPQ2fObAqSrV69mTb21klT4LbcHdeGYvqhauwma1a+OLsOngVryecO7w2vkkhIKI5aNRVtvX/jPm4j6nYYjp+RI+VZICaJgjcVXpmKQsRO6qOowX/hk5Qjs0GxmfUySjqSJk+JTZB6qOIVHgKQ8S0FyBgcHw15I6ffdHtQpgzpj8PQd+O2KBk1rN0SU8Qbig+bin9Vb4VTaVaRfMsKjcU2MXhmFq9p9qPLW37HxgBb/vZUOpUqFh7mFNtH6MD+T+ciXXpmG/sZ2TAEapuyJlar5OKiNYr5wS9uls01E5YHwddeS1NRoNIiNjUVAQACT/rZKUuE33B7UddNHYcqqSAbbqgnDMPKHQBTkZ2KV30h0+LATOnt4YNS0pfjtMZ1SiKjAhejc7kN4eHphwqLNyHpqnTZFARzLr8xEH8MH6M41xueK3liu+gHHtUdZE07+cJKalgEaQiVJcUk2UqVSieXLl7PgkuTkZBZcYu+7uD2ohYUFKCh8LhVzcp437DmPs/Ao+/m2IDoLcp/gYRYjV9hV6vKnh2kIvunPgoJbK6uyaCIK3CCzUVGwRiqDVIzKj71A0XUEKcdxLAIqMjKSGfXLcz+61u1BLZWwcu5MvK+F3/kRaKuqho8V7TGD+wahmmCmjafyZ1wSTgFE8tsLkEZFRYEkqXCsPEsZ1HJCKVxOitHum1uYq5L856Stb9GsZZFFRUEcZuaWLE9lif1agpSae/Lb79u3z2GQ0nvLoAqk2bnMzL+P0IwA9DO2RjvFe5jE+eKAJpxJEQrmcJU+5+s+EqFPSpCSJC0tAup193jVcRlUOwF9kHcPezK24CNjK7RXVMd/uC8RrT1QbHgvCup4VcG70jGhT7p06VKnQEplJYNqI6i5BTnYfXMz+hnboIPifUzhRuIwA7RovI+rKUav+6DI3KRQKBAfH8/MUNT8v+4ae47LoNoAKn83gWnw7ROqY4pqFGK0h0oGpLkboAQbNe+7du3Cxo0bmaZPktUeCK25RgbVClApWmnRpSlozVWFj9IDB7QRLEyuyLcuLU+RNVBYcw5p8xSmt2LFCmbUJw+UNdfZe44M6mtAPf3HEfQ3toWXoiECNOvYGCB3BpRAI0gp2JkgpW17vU22QCuDWgaoTwoeswwerZT/xARuGItUIu+Ru2jxZUFkKUkrClL6HRnUUkDNyL4BX7M3OilrI0C9lgUckxQtq/LcZT9JTuqTVqQkFcpWBvUlUGkA3JAkDwxReLLxRBQgIuXAEKGiy7Ok/icpTjQIz9/fn0m3imjuLZ9ZBtUC1PBbO9CCq4wvlH3B8UqmMFkWljuukyZP5qfw8CInBkHrbMWptHKWQS0GNfRWADpwNTFLPQFKPoFFy5dWYO60jyClMU0UAbV3714WXEKR+m+iDGRQARCk7ZXVsV69FELQCF88rv1NVIoYfpMgValUWLZsGcLCwhzuErX1Hd0e1NCMQLRTVoe/ehlr6ovS0DguwYKtFSKG8y0hpSbfURFQ5Xk3twb10K+haKH8B9aqF7FxSO7oXXoZHgFSau5JkooBUnpGtwU14/E1dNc1xEiuPxuLpOed5/57GQaxbpOSRIoTQSoWSSqUlVuC+uuTmxhgbA9frg/UvFpSozmFinP0kiQpBZRs376dKU6ODtMr7/O6HagFzwow66cx8ExoWJJUtryFKPXrKSkE5X46fvw4G9dE0IrtndwO1NjfovBhQg2WK54SN4itQir6eagPSs08afcUnf8mbKTWvLNbgXrmQRK8+AZYrJ6JM/w5GdLkZKYwEaRkLxWjJBUgdhtQKX/T0CQPlkSMEjnIwSXJrC8qBUgJVrcBNeDGCnRIeB8HtfvYUGXhS3XHJTX3oaGhoKEjlCBCzJJUqB+3APWP3Nvw1NXDXNV3SOPPum2TT+5P0uwtIa3o4BIBPFuXbgEqTZrQW9ESCm2825qiSEkiUKnCaZQoKU5SgZSgdnlQ9fc4tOGqIkizmfnxbf2SXeF8ApIqeu3atcwERTZSsWr3ZZW3S4P6tDAfY9M+YeOcKKbUHeNKBalJ3iYaPiJslwWEWPe7NKgXHqaivao6gjSb3DK2VJCkBGlISIho/Pb2fAwuDer6a/MxROlZITN82FP4zrxGgJTMT6Q8ic0lauu7uyyot56kozffAtvU/myKQ1sLRsrnE6RkdiJJSpCKJQKqPGXqsqDSzMYfKdowU5Q7GfcJUgp4pgF4rgIpAe6SoNIksx8ZWjO7KQ3Os3Y+zvJ88WK4liCl56BBeGSComATMTyXI57BJUE9/vt+9OCasMm/aG4lRxSU2O9BfVAKLtm2bRsDVAreJlvK1CVBHZ3aH18p+7EZRNwhap8gpXykpDhRsjKp2UitAdblQL2adQE9tU2xW72dpYC0phCkfA5BSunHyW9PmZ6dlU3vTZeRy4EacmsrU6JoXD7NWvemC9iZv28pSQlSV2vuLcvOpUAtfFaIsWmDMFX5tcubpAjKiIgIJknJb++qklSA1aVAvZr1E3pom2CPZjubCkd4SVdaUmAJafM0d9PKlStZcIkrS1Kh7lwK1N0Zm/Gxsh1Lx+OKzT6ZnygS/8iRI8xWSuuCSUqoUFddOgXUrD/uICIoCHv2hiMiLBQ7gvfCfPYC4o8fRNCO7dh35BRoDuecuxk4FBWOoO3bEXrgJB7lWySCAuDn54eWLVu+uLOMrfzCPIxLHYRpyjEsC7SrVRgBSdKUpmmkBLrU1NO2q71nWe/jFFDvp6dj7sSJaFmvKt6u2QgTvpuLvcEbUPfddzDoq5Fo36QeJizfi7T4UFT5W2V8PmY8Ojavj/4TVyLn+dxkNoFKzX5XTV0Wzkd5TMt6YSnuFyCl+e1pCnFXMuRbWx9OAVUQepv9BqPH6IVs81fjPjRr4oHbACLnfY66nT9D/OE9aNWuL5sEN95/Aqo06IU/nwpX2yZRQ24FwFPREEe1h5n91NoCEPt5BCnZRak/SpCSpi/2Z3bG8zkV1NWTBsLrq9mMvFumw6hXtTI6efVAi8bNsTZcgcuaSLz3zjvw7NEbzZo3R0C07jmlNjT9pO0vvDwZI5T9WdJdVzHyy5A+zwHmZFAHwLMY1Jv6/WhUuz5WBEch5cINBuT549tRt0FrbAoMQIcWTRAQm2oXqPfy7uBjQ1uWjY8mIXPGF13R9xSGjVBwCWV5dldJKpS7U0FdMtobbYdMZfBdU4Wgdo3mSL77vBOadMgfNep1xh0A0ctGo3LtrriRmVcCq7XK1Mnfo9GTa1YyU7PwclJdkiQlZWnz5s1u3dxb1p9TQT1nUOCkxszAy/r9Og5Hx+L3rOeq/d2Mi4iOOYmHhUD+vQyEh4Yh/c9sm0Hdnr4GA5WdWB4pqYf0EaAnTpxgycooYRlBa1lh7rruVFBLiLNzxVqJOiK1L2Zw4yTvjaIA5wMHDrBJb8klKkNaQX1UO/ksucwaUH95ko5PDB2xUb1S0kEoAqQLFy5k0lSG9Dmk1IpIXqKq78ahq7I+YrUxTOOXYtMoQLpo0SLmEnV1v709dSR5UNddmwdfZW9oeI0k80mRNk8pH2n6cHcILrEHUslL1GfPCjH+jA/GKz9j/VOp2U+peadY0iVLlsiQ8i829S8DLWmJev3xZfTkm2KVer6koqXIRipEPAUHB+PUqVMuH6b3Mni2bksa1AuPUuGlro9ITahkQBWGiQQGBpYAKuyztfLc6XxJgxp80x8+nAcU2gQ2YYTYK46ApP9169Zhx44dbN2dIqDKUz+SBnXJFT98puxR7N937nzx5Slkupb6owQlQRoUFMRcojKkr+6XWpa5ZEHNLniMMakDMJubxKaDFLMiJUBK2fQIUgrTkyG1HlICVrKg/jfnFrrxjbBNvUHUHilq6i0lqQypbYAKUlWyoMbcjoAnVx8R2hDRTrArSE2SpKTdy5DaB6mkJeqOm+vwkaItNLxWlIZ+au7JwxQQECA396+xkQpS81VLSUrUZ3iGdT/PxzBlT2h5XnQJeglQmlyMspYkJCTI2r27gvqkMBs+SZ2xVDVbdP1TcolGR0czbxOBKvvt7W/uLSWsJCVq5tP7GGjqgNWqBaIClSA9fPgwKLiEvE2C98mywOV1+8CVJKin7xxBL665qAbyCZKUwvRIkrrjSFFnfoSSBDXmt3B4KRrhtDYORhFMX05QUnO/YMECGVIH9EdLA16SoM68MJrNdMLx3BtNhEbmJ4L06NGjrLknxUnuk9rXtJcGp+U+SYL6VUpvTOfGvtGZToRAEuqLnj59mmn3cp/UOZASsJIDNbsgC77JPbGQm/bGhp4QpCQ5yW8vJIUQwLWUAvK648CVHKim+2p4qRoiWrv/jQw9ESBdv349M+aLoamnSjQmJuPs2TNINBbN0kcfiU5vQEpqGs6dO4czaakw6oummOR1eiSnpLL9aSlm6HWOA8pZH6fkQDVn6uGhqvtGQLWElOJJSdMX3KTOqiBr7ms0mRATEYwZM2djf2w8TEYD9AYjEvUa+C+bgzFjxmD2wtVI0BqRaDIiKdGI4E0rMW7saMxd7g+FWge9yBOuSQ7UnRkbMYjrggQ+vkIVKYKU+qAkScUEKa8zIDVZh0mDu6NSpUr4ZMICmM+eR7JOgbEDuqJmg1bw/epzNK1RDV18vkFiSgoWTxyKajVq45Ohw9BvwCfYFn6UwWvNR/GmzpEcqN+d98VY5VCWWrKiQvtIahKoBCnNOiIWSUrQGEzJSDgUjO5e3TDS91M0bN4FxwxncTR4KSr/9R9YsTsWNzPSEblhFqpU+TfWbNmExlX+jlFzN+PilSsw6bXgOBV0TjIrOQpsyYHqd34ExiqHIJFPRkWASsEl9L9p0ybRQUoQmFNTsG76V2ja8WPsCwtCwxrVsXTnEexbOxlvv9cM+0/rcebsOcTv34Y6NWri26lTUOPdOlgbdgJnzCYYjCYYDHoWM+EoqJxxH0mBeuvJDfTVtUKgen2FjJGipp5sozSMmWylJFXF0CcVQNDpjTAbVRjerTkqV6uFLl06oMpb/4ceX36P6N1rUPmtf2JlyHGkp19D8OIJeLtqQ2wO3IQGJFHnbcFPly5Br+EQr1CK6r2E97NcSgrU69mX4aGui1DNj04HlQz5MTExbKpGYby9mCClSjQlmxG9cy3q1ayL+Rt3Iy7uBFZM8cU/3m2KsMMn8HW/jqhZvyWGDRuMBrVr4dNvFyMlJRlzvhmEd6vXwcDBQ+HdbyA27D4Ic6JJ1DmuJAWq7p4C3uoWOKCJRKITZ+SjPijlyaekEBSuR02/5dctlnWD0YjYA+FY478VKn0iUlJToY2PxaqVK3HklAYmzWmsXjwXfn7TsCZgNwyJSTAYTEg08dixYRWm+flh2foAKDWy1l+SR8qelZdzT636+XsMVXgx/76Od85gPktIKaueqINLtFrojSakFNtCSeroDEakpKTAoNcVraemIi0tDSnJScxeys4hO6o5pWi/OVm2o9oDp+U1L4O69toc+Cg8mFnKGaASlIIkPXnypLghFbmW7uhWRzJNP0X1z704EaOUA4slqmNnBCEPE/VJqbknSMXgcXJ0ZUv5fpIBNbPgPvroP8Am9SqHTs9DChJBeezYMRaVL0tScbpTJQPqo8KH6KVvXpwH1TF5+gUtnjI70xz3cg4ocUJKLYFkQNU/UKK3+gMc0EQ4JBhFkKQbN24smcxBrNq9lJtsRz27ZEDdeWsD+iS0Aq8t/6hTgpQUJ/I2bdmypSTdjqMKVb6P4yWzZECNvB2MnopmUGm5co3jFyKgSJLSrCMErNAFkAFzPGCOKlPRg9q6ZWtmsVpw5T8YrRwEvdZgt49fhlS8IL4OaNGD2qZVWwbqoKQOWKaajRQ+zW4vEUnODRs2sOZeTBFQr6sk+bgElCkClWyoQ81dsVg1065xUqQk0YQOUVFRLCpfhlR6klX0ErVzGw8YHnLoqWmGaA0NP7E+eIJejvqgZCOl+Zuo6aeIKLlPKoNq6QEt9zq5UD3aeiHuXjS8uIY2j+MnSMmAT+PtyTUqjxKVHqBCt0f0EtWjrSeUD47BU9UAJ7XHrE44QZBSUAlBShKVtoWXlpfSA1bUoE7zm4YOrTti8bUpGKP2KQ5Geb2P3xJSCtOjPqkMp/TgtKwzcYM6bTrat/4Q4y/4YLp6HFKsmOKcICU4KQeUDKm04ZQUqCRRv/tpOCarR742cx9p99QXpWx6BKnc3MugViq3pmTFDX6YMQdv1/8bBiS2QzC/5ZXDTwQoyeMUGxsrQ+pi8aqibvrnz1iASrUqwYOviyO66FJz9ZOpiSRpWFhYSXZnObjEdSSp0PyLGtR5M+ajUu1K6Kqri0O6ff8DKkFKknTr1q0gSUrbZCsVXk5eug6wkgD1C11vqPSqFzKjEJQU8EyQUhSUbMh3HShLEzCSAHWy/kuY9aklwSilQSpLUhnU0tSeClGmhKZ/mn4UkvRmBqoMqWsDWZo0pX2ilqhzZ8xjfdRgwxYkE6jFipNlcy9LUvcAVxKghhiCkWpKY18Vx3GIiIiQFScXMz+VJUmF/aIGdc6MuUyiRplDwZ1WsWx6Go2GuURJkpKElf/dowwIWDI7Zmdnl9YVLXNfhfRRZ0yfiVr13oeSV2D9uvVspOiZM2dYJhDKBiL/u08ZmM1mltklJyenTChLO1AhoE6fPh1/qfQXNGvSDM2aN0OfPn3QrVs3+d8Ny8DLy4vVf3p6emk8lrmvQkCNjIxkD+ft7Y1evXqhe/fu8r+blgEJqL59+0KUoJb5mcgH5BKwsgQqRKJa+SzyaXIJlFkCMqhlFo18QEwlIINqZ23k5+XiYeYD3L17D09y8/HsWQGyHj3E3Xv38az4nk/zcvHoUSbu3r2PQjt/R76sqARkUO0hIfdPTPbtgyYtWqNz517YG5eKuNDlaNSoKTp2bIN+vpPx673HiFgxHg0bN0eXju3Rw2cMzty8a8+vydcAkEG1B4PMG+jRoj5m74hDfl4+nj4twLoJ/dHBZwayHv2KNu/+DbODT2D1uL7oOXIFcrP+RNfaVfD5wlB7fk2+RgbVTgYeZaB/hyYYNGYWQvcdRPqdRwie7YvGnQYiKHAtalV7D6GKNGyZ8gladhuOkJ2B8PywLXbFnbHzB+XLZIlqDwNZv2Bgx4Zo0bEnvp44FYk//4G9i77Gv2o2hu/IsQg6EM/u6j+xD2o3+RAereuiVf9xeGLPb8nXsBKQQbUHhMzr8GxaF8v2JZVcvWpMH3QY/H3JNq3M9mmHjycF4umDK2j6XjWsiNS9cFzesL4EZFCtL6vnZz6+jeljR2DPqedN+f4NczBpwbbn5wDYteQ/mLkmku0LXvAtRs7yR94LZ8gb1paADKq1JeWA8woLC0pMVw64nVvdQgbVrapbui8rgyrdunOrJ5dBdavqlu7L/j8aExpUW8dPFAAAAABJRU5ErkJggg=="}}},{"metadata":{},"cell_type":"markdown","source":"On high-level, the higher the AUC of a model the better it is. But sometimes threshold independent measure is not what you want, e.g. you may care about your model recall and require that to be higher than 99% (while it has a reasonable precision or FPR). In that case, you may want to tune your model threshold such that it meets your minimum requirement on those metrics (and you may not care even if you model AUC is not too high).\n\nTherefore in order to decide how to evaluate your classification model performance, perhaps you want to have a good understanding of the business/problem requirement and the impact of low recall vs. low precision, and decide what metric to optimize for.\n\n* AUC is useful as a single number summary of classifier performance\n* Higher value = better classifier\n* If you randomly chose one positive and one negative observation, **AUC represents the likelihood that your classifier will assign a higher predicted probability to the positive observation**."},{"metadata":{},"cell_type":"markdown","source":"# ROC-AUC (Intuitive Explanation)"},{"metadata":{},"cell_type":"markdown","source":"Above ROC and AUC sections must have given you basic idea of what they are. Now **let's get into the intuitive part and see how exactly ROC curve is drawn.**\n\n\n# Ideal Case (When classes are perfectly separable)\n\n![](https://miro.medium.com/max/660/1*Uu-t4pOotRQFoyrfqEvIEg.png)\n\n![](https://miro.medium.com/max/404/1*HmVIhSKznoW8tFsCLeQjRw.png)\n\n* We start with threshold = 1 (Visualize the steps in the above diagrams as we proceed)\n\n  TPR = 0 and FPR = 0 since TP and FP = 0 because at threshold = 1 every point is predicted negative\n   \n  That means right now we are at bottom left corner of ROC graph i.e. (0, 0).\n\n   \n* Threshold --> 1 to 0.5\n   \n  TPR starts increasing as TPs are increasing.\n   \n  FPR remains 0 as FP = 0, because classes are perfectly seperable\n   \n  Thats when the graph starts to go perpendicularly upwards i.e from (0, 0) to (0, 1).\n   \n  And at threshold = 0.5, we reach at top left corner of the ROC curve\n   \n\n* Threshold --> 0.5 to 0\n\n  Now, the true negative labels start coming in. And if they are predicted +ve, this would be a FP.\n   \n  And hence the FPR would start increasing from here on with no increase in FPR.\n   \n  In graph, this would be a horizontal line from (0, 1) to (1, 1).\n\n\nA simple would mathematics is required to calculate the Area Under Curve of this square which is --> 1.\n\n**This was the ideal case.**"},{"metadata":{},"cell_type":"markdown","source":"# Practicle Case (When classes are overlapping)\n\n![](https://miro.medium.com/max/634/1*yF8hvKR9eNfqqej2JnVKzg.png)\n\n![](https://miro.medium.com/max/425/1*-tPXUvvNIZDbqXP0qqYNuQ.png)\n\n* We start with threshold = 1 (Again visualize the steps in the above diagrams as we proceed)\n\n  TPR = 0 and FPR = 0 since TP and FP = 0 because at threshold = 1 every point is predicted negative\n   \n  That means right now we are at bottom left corner of ROC graph i.e. (0, 0).\n\n   \n* Threshold --> 1 to 0\n   \n  Now things here won't go as they did in Ideal case. Because now somewhere in threshold range of 1 to 0.5, negative labels would start coming in.\n   \n  **TPR starts increasing** as TPs are increasing.\n   \n  After some decrease of threshold, **FPR also starts increasing**, because classes are perfectly seperable\n   \n  When FPR starts increasing from 0, thats when the ROC curve starts to bend from the perpendicular line (0, 0), (0, 1).\n   \n  Our main our is to find the sweet spot where the FPR is not increased much (subject to the problem) and TPR is as high as possible.\n   \n  Initially as theshold is decreased from 1 to lets says 0.6 (just for explanation consider 0.6 is sweet spot)\n        *    Increase of TPS would be much faster.\n        *    Increase of FPR would be slower.\n   \n  As the threshold is further decreased towards 0 from 0.6\n        *    Increase of TPS would be slower.\n        *    Increase of FPR would be much faster.\n    \n  With this we would obtain a graph something as depicted above. \n    \n  One can use calculus to obtain the AUC for the curve but it would be somewhere between 0.5 to 1."},{"metadata":{},"cell_type":"markdown","source":"# Practicle Objective\n\nOur objective with ROC-AUC curve is to find a threshold value that results in **largest AUC possible**. With the above pictorial impression in mind, one can imagine how the graph would like in a good scenerio."},{"metadata":{},"cell_type":"markdown","source":"# How to use AUC ROC curve for multi-class model?\n\nIn multi-class model, we can plot N number of AUC ROC Curves for N number classes using One vs ALL methodology. \n\nSo for Example, If you have three classes named X, Y and Z, you will have one ROC for X classified against Y and Z, another ROC for Y classified against X and Z, and a third one of Z classified against Y and X.\n"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}