{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Laplace Log Likelihood\n![](https://i.imgur.com/tEIZvli.png)\n*Image Credits: https://en.wikipedia.org/wiki/Laplace_distribution*\n\nThe evaluation metric of this competition is a modified version of Laplace Log Likelihood. Read more about it on the [Evaluation Page](https://www.kaggle.com/c/osic-pulmonary-fibrosis-progression/overview/evaluation).\n\n![LLL.png](attachment:LLL.png)\n\nThis notebook explores the functioning of this metric with some examples and visualizations.\n","attachments":{"LLL.png":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAASwAAAB7CAYAAADOgb2OAAABQ2lDQ1BJQ0MgUHJvZmlsZQAAKJFjYGASSSwoyGFhYGDIzSspCnJ3UoiIjFJgf8LAzsDBwMUgyiCdmFxc4BgQ4ANUwgCjUcG3awyMIPqyLsiso0IrS7ZUvX0VdmJu0yWPk+8x1aMArpTU4mQg/QeI05ILikoYGBhTgGzl8pICELsDyBYpAjoKyJ4DYqdD2BtA7CQI+whYTUiQM5B9A8gWSM5IBJrB+ALI1klCEk9HYkPtBQEep9S8QAXXcCNzCwMCziUdlKRWlIBo5/yCyqLM9IwSBUdgKKUqeOYl6+koGBkYAa0EhTlE9WcxcFgyip1CiCXPZmAwWgEUZEWIZR9iYNgrzMAgiKROvRLoLVcGhgNxBYlFiXAHMH5jKU4zNoKweYoYGFh//P//WZaBgX0XA8Pfov//f8/9///vEgYG5ptAvYUAxk5g/vyMwEsAAABWZVhJZk1NACoAAAAIAAGHaQAEAAAAAQAAABoAAAAAAAOShgAHAAAAEgAAAESgAgAEAAAAAQAAASygAwAEAAAAAQAAAHsAAAAAQVNDSUkAAABTY3JlZW5zaG90QdvVvQAAAdZpVFh0WE1MOmNvbS5hZG9iZS54bXAAAAAAADx4OnhtcG1ldGEgeG1sbnM6eD0iYWRvYmU6bnM6bWV0YS8iIHg6eG1wdGs9IlhNUCBDb3JlIDUuNC4wIj4KICAgPHJkZjpSREYgeG1sbnM6cmRmPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5LzAyLzIyLXJkZi1zeW50YXgtbnMjIj4KICAgICAgPHJkZjpEZXNjcmlwdGlvbiByZGY6YWJvdXQ9IiIKICAgICAgICAgICAgeG1sbnM6ZXhpZj0iaHR0cDovL25zLmFkb2JlLmNvbS9leGlmLzEuMC8iPgogICAgICAgICA8ZXhpZjpQaXhlbFhEaW1lbnNpb24+MzAwPC9leGlmOlBpeGVsWERpbWVuc2lvbj4KICAgICAgICAgPGV4aWY6VXNlckNvbW1lbnQ+U2NyZWVuc2hvdDwvZXhpZjpVc2VyQ29tbWVudD4KICAgICAgICAgPGV4aWY6UGl4ZWxZRGltZW5zaW9uPjEyMzwvZXhpZjpQaXhlbFlEaW1lbnNpb24+CiAgICAgIDwvcmRmOkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgrHERmhAAAuuklEQVR4Ae2dCbwUxfHHW03UeKBokIBG8AA1nhyCciOKIuFSQCAETzyjKAYECSISVEQF8QBBUVHBIyAoGhRFvBFQIyEGiIl4R/BMDDExuv/+1p/aT++82X27sPt2l1f1+by3Mz3dPT013b+uqq6e2irhyRkZB4wDxoEy4MDWZdBGa6JxwDhgHBAOGGBZRzAOGAfKhgMGWGXzqqyhxgHjgAGW9YEthgP//e9/i/YsX3/9tfv+++9T7v+f//wn5dxONp8DP9j8KqwG40DxOfD44487AKtHjx55acz999/vFi9eHFvXb37zG1evXj25tmzZMvfwww+7PfbYw33wwQeuefPmrk+fPm6rrbZyr776qvvjH//ozj33XLfNNtvE1mWJuXHAACs3flnuEuMAi9w333yz23HHHd2ZZ56Zl9Z98803bt68ea5BgwZuzz33TNa5Zs0a99FHH7m6detK2sqVK91VV13lhgwZ4tq0aeM+++wzd95558m1vn37StqGDRvc1Vdf7YYPH+5+8AMbbklmbuKBcXATGWfFSoMDCxcudH/5y1/cxIkT89agJUuWuF/84heuW7duKXWOHz/e1a9f3/3whz+U9JkzZ7of/ehHrnXr1nK+++67uw4dOjjSu3Tp4nbaaSd3wgknuNdff93NmTPH9e7dO6U+O8mdA2bDyp1nVqJEOPDpp5+KdHXaaaflVeUCgLp27ZrylNijnn/+edeiRQtJX79+vah7TZo0EfVPMx900EFy+PLLL2uS69+/v7v33nvdxx9/nEyzg03jgElYOfLtf//7n1u6dKnM6u+884777rvvpAZm08suuyzH2oqbHXUqG79h7DH8pSNUo0WLFglPTjzxRHf44Ye7W265xb399ttu7Nixbu+993Z33HGHe+6559xZZ53lOnbsmKwK0EH6QJ2iLf/+979FMoGXL7zwgnvqqafc6tWr3ahRo9wBBxzg7r77bkmbMmWKe+ONN6SeQw45JFlfePD++++7P/zhD1L+q6++Sl7q3r27A2jSUZy9CSkJ4tkgBZ+f/OQncq7/atWqJYcffvihJsnz77LLLsIjJDejTeeAAVYOvKOTjhs3zv31r38VVYDBdcQRR0gNderUyaGm0sh6xRVXyICurDWoQdiJ0tGBBx4o4LJ27Vq32267CVg1btxYJBDUtVmzZrmjjjrKLViwwK1YsSIJWH/7298cbRgwYIA7++yzHcCHrWfgwIFyK8pQD0ZspJsXX3zR7bXXXmLErlmzpgAibdtuu+1SmgbwzZ07102fPj2Z/uMf/1jKkoCBPFd66aWXxCal9/r888+lClTCkHbYYQc5XbduXZjs4BFGfAOsFLbkfGKAlSXL/vWvf4lxlZl68ODB0nlHjhwpHbGyTgjAMdMjaRx55JEOQyz1FArk/vGPf8gAp82nnHJK2ifs16+f69SpU9rregHpMRNhTP72228FxJ988kk3dOhQkWoog+SFygZgQA0bNpRf3AAAq5YtWyYBTKWgww47TPJgK+KPSeGJJ54Q4OrcubNc499bb73ldt555+S5Hjz66KMCVqhnI0aMcAApNq4LLrjARSUiLZPpF3UQ6XDYsGEVsm29dapVRSXRqItDjRo13N///neRyOMkuAoVW0IsBwywYtlSMZFlbgbUGWec4dq3by8ZkACmTZsmdo199tmnYqGNKUgFqDSXXnqppCxfvlykDVaPCkEMDgCBAZKJ1N6SKU+211DPkDhZHdt1113dn/70JykKIKGyAVyQgpHyMzRE//nPfxbQU5cBKeD/HXzwwQL4P//5zzVJfgEiJoCQsC2hfqKCwXNWD7knqicS1+WXXx5mz+pYVU+kPSWeEWLyCQkeQHpdr6lkxmSCdGi0aRwwwMqSb6gkDAJWf5R0pmQZPBNxHbBTgGAQZ7KhZKor22sswevqVboyqFgM8MoIAGT1Kx1hrwEQjjvuOHEFIB+gjFSlEhGqIPwDjLD7zZ8/3x199NGOlTXoyy+/FFeCVq1aVbCXvfvuu5IHPyfqUKL+KGC8+eabchnpEbAKScEkTMvmGD7R1lD9U9cGVQ21Hp4D0uuajpRG+SiQ6XX7zY4DBlhZ8Omf//ynAA4rRKEvDXYNOiH+OkoMCqQJJBxmdkAKaQMpYfvtt3cMKIzIjRo1kpkWwzKzLx0ZFeeYY44RW8vTTz/tPvnkE1GZGDDUw6BRwiERm89Pf/pTxyCHMFxTH2oUDo2oQJkIUNEBnikf0mMmwELdhXRlDfDC4I2BfdtttxVjOvdq2rSp8EUN1hjRldTedOihh4qkpuDARKGDHAkMPsJb1FQk1yjgKrj97Gc/06qT0h4Sca6k6qBKx1oeOxjvPco/2giF74pz+gXPqyojaUa5cyBVAc+9fLUogZ2E2TxcUXv22WfFqIzaoSDGbIv9BgMrA++ee+4R/tCpVZ1AwmCJW+tEOsFvhwHI34QJE2SQY7xl9YwByMoU6iPACaHysGwOID744IOShkQ1ZswYGSisZAJkKr1Ihph/F198sbvrrrsq/cPWlInwW8LBEgM4pIO4WbNmco5qioS57777igqtW2gAWKRP1ENsYBDgfeuttwrgMvgxVAN8++23n9iRAG/sSRCSKvzDVqfEPSC1IXHt9ttvF9UxtNch5fH8p59+ujiDavnor4IxYBsl7IOAs+bhXkw0mAyihv1Vq1YlJ7Zs7x29n507Z4CVZS9gewUSECA0efJkGUgMLGZ8pUmTJomjIIOLGX706NFyiSVxAAzCFQLDrxp/qZNBA/jUrl07OXBxCUCqQc1Sb2sGNRIUkh3uA7Nnz5YVNWbva665Rozb1IFawhaRqiAAhwEbqp/YfHg+XVRASoFw8mSBAp7xvI899pi78MILHa4Axx57rORBOsXZEg/ym266yQ0aNEh8rNq2beuwWQGOCjwqWSpgUAEgSTpuFQ899JC4VSC1YrsKDeSABsAJ4FBnOoLXqO9xCw/wGM/2qVOnuhkzZsj7ZnLhmULCO5774A0PZXvvsA473sgBLzUYZckBDwwJb0dJ+AFYoYSXFhLeKJzwM37KNa/WSboHG0n3fkQJP+PLsZbxM7Oc+wGavHbOOeck/MCXdD+IE5xDfh+b/PmZPOHBInndSyFy7KWrhF+VS5aVxBL450E0pRVeAkr4gZzgF+IXfih5tTGhPNPrftDr5eSvd5UQ3vDcIcH3uPxhHvjHu/CqdZiccky7vWSbkhY98QCU8GCaoH/E0bXXXpvwrh0pl7K5d0oBOxEOmISVw9SFDQppB7tMlNQ2odfwP0Iiwn7FShb2JmZWbE+ojLg6kIaqiPr33nvviUOqqhmoOqyuIT2h9qk9CvUKlQPpCxUEQzT+TGoPQqpgNsfuw4pUqVBoLKdN8AsJTPnGLz5cSkhoqmpr/jgV9/jjjxdXCdRk36O1uKhkcfmTGfwBktkXX3wh0nCYHh7T7jjpKsyD5IaqT/+I0u9//3t57z179ky5lM29UwrYiXBgmys9GS82nwMYiQEKbBiACoZvbDqodqhIqAqAHf5EAAmqHl8YIC8gp/5L2MrY5U+Hxg6C2tSrV6+khzWgh82LcvyhApHGFwMALtRRymOQRu2qDoQxHVUbm1dobK/s2bGhnXTSSUnQrCx/rtdZMKBPxH2todD3zrWt5ZJ/K+SscmlsObQTwzjL6aG9BDuPzr7Yc1QK++UvfynOiKzChUvwODkCOEgPcY6R3AOADCUQVs6Q1LivrqKVA7/y2UZsUsrbfNa7qXWVWns29TlKqZyphHl+GwBMCFZUr2DFMatgqD9IBKycoR6GYIUayQogkkIcWFEH6SFYkYbaovetTIUh/5ZIpQRW8LfU2rMlvHMDrCK9RZb+8fLG1hUS/lM4p+KDZWQcMA6kcsBUwlR+2JlxwDhQwhwwCauEX441zThgHEjlgAFWKj/szDhgHChhDhhglfDLsaYZB4wDqRwwwErlh50ZB4wDJcwBA6wSfjnWNOOAcSCVAwZYqfywM+OAcaCEOWCAVcIvx5pmHDAOpHLAACuVH3ZmHDAOlDAHDLBK+OVY04wDxoFUDhhgpfLDzowDxoES5oABVgm/HGuaccA4kMoBA6xUftiZccA4UMIcMMAq4ZdjTTMOGAdSOWCAlcoPOzMOGAdKmAMGWCX8cqxpxgHjQCoHDLBS+WFnxgHjQAlzICNg8f3x6vrJdw32SVBSYgHyOWMj40ApcoAgJ0Rd2lQiMlMm0gC+6fIQBIUITpkILNExFeYjPRfKGKqe4JwEQoiG3c7lBsXKS0QamEFUlFwIpk6bNs21a9dOAn4S4ea6666Tv2ioKh9jUMJspQN1vr1OdGFCexH9hhcL7b///q5v377JZhGiSsO3+9iGEsY+edEfEA2GqDqEBuPTyQRLJToOATp/9atfub333jvMXtLHRIshbBm/6YhoP926dTO+pWNQJJ3I1gSizbUf8Hluom4Tnen888+P1Orc7373OwkyS8g1IpDTZ1u0aJHMx/ji3vRJgqIwVggiS9RxJfr1lClTJJYB12vWrOkuuugiyY8wQMDcU089VUK1aZmMv0TNiSPfQAkAeumll8ZdLvm04cOHJ3yU4Zza6RmauOSSSxI+CESynB9YwgcfoTmZFh74MO5y/b777kt4KSxB4M33338/MX369ISPNpzM6sNwST6CakbJv3i55qNKJwOLkodgm2PHjpVrPqS8BHElYKiPJC3BQwncStlyI56BtvO3dOnSBHz38QETfgBJkFgPaMlHMr4lWZH2gCC78CkX8mCU7Fs+gnmFosp3gt1CvCfelwb3JfDtqFGjEj7ydcJP8pKHPsqYY8xAn3/+uQT19VG45Zz3zvi67LLLkv2c8UKaj3EgeSr7l1YlnD9/vgDd6tWrJUR7RtQrwYuDBw9248aNy6llzBZEq8lFokTKgYgzWKNGDYcURoguQslrcFOua4xAouJEiXDozEr9+vVLxsgjNBizzyuvvOJ8tGcJQ09cw2222UYkMAKuUqYcI7MQvBQiBiNBZpFECfyK5ElYeaRHJeObciK/vyeffLLES4yrlb7nJ0/Xtm1bCXZLHt4T/c9PzFKEAMGvvfaaSMLE4IS6du0qphOCx0JPPfWUxGUkHaLvovFQliAsEOMFLQEtJhuzSyxgUXDx4sXJB5ozZ45UXk7/GAwwOFvyM4h77rnnXJ8+fbItkgxmyn0QqyHEZOICEnW4R48eybpUTCZSc0jknzFjhojkYeiuqVOnuhUrVgjwNW/ePCwix4BqmzZtKqSXQwIqMkQAVCX6HCozA+PQQw/V5KR6YXxLsiRvBwBIHDEWoIMPPjjlMpMHAgzvgiCxEGHqlIjcjfo4b948SSKoMKoiE7iS5uea0r777ivvHJCsjGIBC3sJOjH2KxCQSMLRDpOpYi8uumz+vPiXqRqJooyUNHDgQLd27VpBdKLo8scsQORjL5ZKuCyiHkPPPPOMSFaUIXoykXe9eua8eubGjBkj5SZPnixliMKs9OKLL8oLyiWmH4AChYDywAMPOMLMAz5hXEFeHCDKNXR3JV4uUaEJS6+E3r9w4UJ52V7E1uSUX2Id5gKuKYWLfMLMDDVu3Fh+4ceNN94ogAX/Q6nR+CYsqtJ/9FGI/hqSTsr0T8YjFM0DYGHER4pmfNatWzepNZCfiRzCJhZSkyZNpM9XZryvAFgYnR999FHn9VUZdColkJYNAUIYTLP5A0gyEVIEzGH2xbCHqAnKw1Bmaa97y0CHQYSEhzp06CAdH2aB5hj5mLHJf9BBB7mbbrrJEWkZIA6fCcN2qMJlapdeW758uRxivAQEhw0bJkxv2LChZkn5ZSaB1q9fL7+AKW3A6BgSCwZQq1atRM0Mr5X7MVKUzuCowt6+ISoB7yMEqvA5jW8hNwp/rAsiYYBf7qrn9Fvtw0woIemET6BgSMtoHoIIUyYad1PfsbfPatbY3wqrhCzhM9BBRqhjx46O1TDAghkdsS8T0aARI0aIhJUpH9f0Huny8bAwB5Fy0aJFzhvSpS3kZ1B7Y11SFw7tHt5ALsCmEZd1WXblypUSpBQ0Rxc/4ogj5NZcB+AI9Z4tIRVgX4IIOQ/QswqCHQYexBGrOAxW7lWnTh15FuxWgGpICr716tULk4t+7BcT3IYNGyptByqydtxoZlY6mWCYHDp16iTqM+8rk92wXPiGewETH2ptgwYNRAqh74aqfpQfuZwjlTBJ1qpVq0rMAdF+rOdoT0qapuf6q9pTumePulLo2ENAyUQpgMVNZs+enbLEiVqDARn1CdBKp6KENwltE2F6rsfr1q0T6QoJCwM0OjdqHsQSK+CJ8Q7ERmqCUF3J37RpUznnn6I2nQiwwlBOnkaNGkkeAARKN8PLxci/NWvWSAr1IblBflVEOpOcxPxTOxYvhXvRVkA3Sqp+h7p/NE8xzrExYB6ojHhXyttoXn1/LI+rRMsgTJef8uXCN2w+ftXYtW/fXh579OjRjsWf0CYX5Ucu5wC3ujDkUi7XvDqBRicnBRmkYVQ7hAlMM6GUpXlUGFFhQdsAxpAH1TEkHXt+1TBMrnCcAlgMfkAhtKdQAvUQwMLewuqCSi4VatuY8Mgjj6S7lJKOVKSrQCkXNp7oSgIvnRkbkEF3ZvUi7OzNmjVzW2/9/9qt2pWwC0HMBkg1MLlnz56SFs3DNQgDeLakAw/AUgIkdabQtPBXFwGQVAB/VkfiDJ90GJ41bkUxrC88xgjKc+hzh9fydYxPGZJkZZROuqKcqtEq3ZKG5J6pTxWSb9w/X0TfxKaD9AxNmjQpraTJdQbvnXfeKavDmfoNeSHUaUwbvIdCkoJNFDyYkCHABvDEAE+eELBUncTeBS8QOkJiQQqifEhq19V3HV4Lj1MAC3sKgBQlZjgGJsZSjNqdO3eOZkme8xIAN21A8kLMAZJbJsCic8MMXQ1TAOvevbvUFkpKoD0MUokLdQp0R1cG0fv37y/OaxRE4oIxLKUDDGoIRKXLltQOEwJEaGSPq0dfBi4jqEMKutG8TBh0fjpEnLTKAgMS3gknnCCdmE6PYyoDn8GCFARoIKFRB46m/DLx8MwY9JEGtGMyCFCXec/YzdJRJiBKVyZMp7OiMoUSMdd1wgjzhseF4Bv9BfPHdtttJzxBLcWlAlsKqj6qK+YQ75PnmBDhFf0Juxv9DrcStc8sW7ZM8pNOXvrUkiVL5Ld3797yKPRBzBrwgD7POwHQuC+uNEicABL1Y/9p3bq1tIXCtIU+y4QKWKjxO+RRPo9RaSHcD7AJK9FPuD+CRsuWLaUfrVq1KgnQPBv9lsU6JmKcr3E+BbS0zeSHeL6QVJpTW1Z4LTxOGt2Z9WE8q1wgePRP7So0AMamI3RaPMW9E1mlf7169UpXjczkdBZWK1UK4aUBSmqvovNDDCTaDcFkwBVvajqQSkLqoQugkgbQMNCZEaifgUrnyYZQ6ehEDDxVV7IpRzsZnJTD7pWOAGTyweuwTUwCzz77rMMzXoEFyRIjP2opPi7YNwAnyjEgATcAmckIPzEGEwNFfWeoCz4ziOBZIUnVSd5POttH3P0LwTf6EYs3M2fOlP7DPSZMmCC3B5zgFwBDH2LFC/uRd3gU+y4qO/xFep84caLwmL7Mu0G1hc+MJzU1IJkMHTpUyqIeolrzfgGtLl26SH9lwKrtl/K42ED4MvFeEBIAsnBFWjLE/OO5mKBZGc8kEavhPKq2wQsW21g5VymLCZI+D1Dz7hg/mFjwKGBMQeqqwMQIMaHS15kglRYsWCDCQnQiBuggtdvSL+GN2onlov+XlLBQUSBd0pSTmH/MTHQ8BYCYLHlJUiRmxlJiYIHa2tlVhcNnTG1BDEQYjfs/EgzGdQCODhoSnYs8iujMCiNHjhQ7VKaFBUCSlUaIWZPVwQEDBmQU/cP70hZml0zSGB2ZehlADBLKoDIhLSJt4sqBZADBCyYTQIDOAQFSzITYUyCeFRUMcOM9cw1gA9CY0bE5McHgClIooqPqqjCSOoOQzp8t5Ztv3BfQOf300wWsmbhUBUci4N3CUwYlEwV9BdBgRRhDMuo/vASUBg0alFyMQDrmndB/1WGSCYLByzNA2LaYJJFY2BLDu+R9YwpAEoFXABzvkYGLTx4qI22knmyIiYn6Abk4SZ62q78U4wWQBKToGxAr14Co2iMBDtIQICB4wHgZP36887sxpBy4cOWVV7r69etLHvox56wEI30BfoA/dUZNAAgjmHpUNdaxDSCmLMh4dCxb8jOcbOsIH8DPFgnvyxEmVTgmj2dIhXTP/IRuI9CLvpPIlgTfuTVps359B0puS6isIp7PD56Et7kl/EyfNjtbM7R92l4PYsn8HvgSXmWUuvzsl/CGW7nmPehlK4yfGWUbULJACR7km2/KJ+0rfhJK8gX+eFBJcgHewrdZs2YlaIcS21A86Mupl4QSQ4YMkWP6F/nZmqL38YKAFpNfrpGHfuilIDm++eabZdsV7x2iTbfddpsce6lO8ngQkfPwX9zWHA8Q0h4PvGHWnI9pH8/sQTttWQ+Osm0sbQZ/wWslyS070XzU7yVU2dam17iflzITfmLTJPlNqoQCiWX2j5ksKqkgzipKp3sc8uiqRJjH74sSlZIZp1DE7KMSYmX3IB+SEGqEGnKjZVA3kIr92xQbB7YopAKdzcmPXY80bGfcH6kSSQHpgBUt7BQ47OnqZPQepXCeb77BJxxX6St86QCbpEp8SBOoI0qoTqhwXKcd8BM1CikGFYZ3gBmE+lDH4S35UCf1XWt/4xqSnJouULWoB8J+jEqJxIyER17eFSYYfA7pB9l8lYG+MHfuXNGCopKMPlO2v7SbZ1GzTFw5pDO1M8ZdJw3JLU5z4dmRwFCH4bESfRN7uUp0ml7WgKUPka9fjKg33HCDqEl06HIgQAZVEHUR1QLROjRoqr0RuwnOvCxDs0qKGwiLHqhpfhYX+0M6UCwHPuTaRuyYDBbsVF5yctdff70MGOyTmD3ClXKAAh6j3qC6MUCZKJkEvFQk4ID6CO+5hm0LFQhQQ81jnynqPV9GQN3EbMF7Y1DyzjBXYOhmJwZuC7SB+6F2YsfEBIFKD3BSb2UEYNFmXZyqLH+xrqNqwz/s5eHiFe1BHUUtjoLcVshZxWpwqd4XljALMrtgtKRzs5KDZFZqRFt5ubpiFdc+QIpnwYbFcTjrMqgYHOkc/OLqK/c0eMaiBzsTAO5MvNNnZXAh9YQSPZMBafQLfuGxSiJxfOY+6n6jxnDKQLSJVWoAT6Uy0jWNe1GGdxUlJiAWViqTcqLlSuGcZ1IeZNMeA6xsuGR5tigOYMzGUI6/YHUC6i3hJZpKuCW8RXuGnDiADQr/KGxERuXFAZOwyut9WWuNA9WaAyZhVevXbw9vHCgvDhhgldf7stYaB6o1BwywqvXrt4c3DpQXBwywyut9WWuNA9WaAwZY1fr128MbB8qLAwZY5fW+rLXGgWrNAQOsav367eGNA+XFAQOs8npf1lrjQLXmgAFWtX799vDGgfLigAFWeb0va61xoFpzwACrWr9+e3jjQHlxwACrvN6XtdY4UK05YIBVrV+/PbxxoLw4YIBVXu/LWmscqNYc+EG1fvpq+vB8HZMgroQG0y9gxrGCr6xmG6UlrrylGQfyzQEDrHxztAzqI8gG8RTD6MtxzdaQT3HXLM04UAwOGGAVg+tFvufjjz/uzjrrrJRAC0TN4XPBGu8wUxP5vjnfINdgrJny2jXjQD45YICVT26WQV1E7wWcCFUPERiWyCVEKoaItvLrX/+6QrQSubjxH1GRCW0F6BkZB6qSA2Z0r0pul8C9CH3uA3hKZBYillx33XUS6t4HBZWoK0QLJto0ElQcIV3NmTNHogYTccfIOFCVHDDAqkpuF/leGNsJW9+uXTtpiY+s68444wyJC9enTx+JuUeMPOLlqcQVbfLLL7+cDPxJuHMj40BVcsAAqyq5XeR7YWw/6qijkhF2kZZatWqVbBXx8I455hg59yHKk+nhAdLVBRdcIEkE+dT4emEeOzYOFIoDBliF4myR6kWKShe+CmN7x44dky3r1KlTStBOLlCeYJ2ESI8S4dUBqOOPP961adNG8iKlGRkHqooDBlhVxekquA9RgkeMGCF2qagNiuChobE9XXOIco3vlUYwDvPNmzfPnXzyyQJyXbt2lUsPPfSQhGYP89mxcaBQHDDAKhRnq7jezz77zA0dOlSkK2xQSEMhPfnkk0lje5geHi9atEjC2BPGPUoff/yxA8xat24tlw444ADXoEEDsXctW7Ysmt3OjQMF4YABVkHYWvWVYo+68sor3R133CE3ZzVQSY3t7du316QKvxjZ7733XlkhjPOvQp086aST3Lbbbpss26NHDzmePXt2Ms0OjAOF5IABViG5W4V1165d2+GZzi+2KVYDURGhl156SYztNWrUiG0R7gkTJkxwI0eOlPLRTF9//bUDAKPbdI4++mgx4OPLxZ+RcaDQHDDAKjSHi1D/scceK3ddvHix/D7xxBMpxvawSRs2bHBjx44VJ9DQ0E7ZL774QrIuXLhQjOy77rprWFQ845G6oEceeSTlmp0YBwrBAfN0LwRXi1xnw4YNXf369R1AhUc7oKSe7WHTcF347W9/Kyt/b7zxhuMPwh7GSiP+WqiaqHysDr799tthcTnGkA+98sor7sMPPxTnU0moRv9w7wDUseulo5122smdffbZ6S5bepYcMMDKklHllu3EE090t912mxs/frzjGB+rKLElB892KOoKce6550r60qVLk46ibOvJRNi5ijkoAWDUW5xfN5UA5tBOV1k93333nXj+d+nSxdWqVUuyA/LffvutuIdo+Vzq1DKl9Atvs9lnWug2b+WXv+P3YBT6zlVYP484efJk17x5c9ekSZMqvHPxboXdqW/fvtKA+++/36WzXxWvhfm7M/azhx9+2LFa2bNnT3fqqafmXDlANW3aNJEq46TRdBW+9tprIoFeffXVIpneeuut0g7y77nnnu7CCy+MlW7T1Veq6c8//7xMbkxkcS4vVdXuamHDQm3BJlOdnBxRQY477jjHyuCWDFYMFL7r1b17900eM0hlw4YNc40bN84ZXNiepIsRt99+u0PiAjABPVRk6sUlpNwJR+H99tvPAcxIkMWisgUs9rQxu2VDDF42+Va3rwuwT3DgwIHZsKjs8yDNbCoBNHyskFXPXOjTTz91K1askBVYwKlmzZpu9OjRIuVde+21MmFQ35tvvplLtSWbVx2K2Z5VLCpLwGJP3DXXXCPL+NkyDiP0li5pRHkBUO+8887R5Gp1/u6777pvvvlGthHhTBvdI4mNDsmbzd+5Eo622AexT33wwQfulFNOSalCnWyj90zJVGYn/fv3F3+9YkmNBTW6P/300yIFsbrElhFWn7ATQBMnTnRffvml2JawQTAjhcvqMITvLiFiY5sYNGiQzILMYMxq0JQpU9xpp50mBmM63po1a6ReVrWeeeYZd8sttzhE9nfeeUdWuHCq3H777aUs9dLhXn/9dfkQHbNlpu9AYQfL1tyX6bPDcnP7V3AO0GfuvPNOUcsGDBggNi7dJ0kfQo2EmPxQ3wD3XIj+Q/+k30LYR6PE/aBDDjkkekm+hsHOgdWrVycXNciEalvKdta9995bfO8YO3ySqKqpoICFiD1//nx5IZ988oks/R522GECIqxO8cI5BzQANQUswOf6668XkGrZsqWoNYjVfNKXztavXz8xjmIAZPUL3XrlypXS6QAqZjQ6y1dffeXorMx8SFgKVujgLOdDw4cPF52cGZal6egsqS8EELznnnv0NOPvzJkzq71kk5FBVXDxyCOPlIls1qxZDikLL34mQYzgONIqYDFZtmjRIm2LuE7/iq7yATYM3r322itt2bfeeku2L9H3lJj05s6d66ZPn65Jsqqp9eyxxx7J9FI9OPDAA8UmvMUB1o477uiQXHgZIDLgcPfdd8t7YAn8kksuSer3dAoI0XrMmDGubdu2DrDCY5u9cdqp6HyA0eGHH54SQIHNvQASM99FF13kzj//fHFspJOSv2nTplI//6ZOnSqS31133SVLtSzX8tE6XkQ6Anzr1q2b7nJK+g477JByHj1hVs2G2KtXXaQ1tgbhL1YZYavKVhrCpgThcsA7Bjh22WUXB5BAGNvpW+neF5MfmsCQIUPEcVYKbfyH5M6OgnTE87DQw8QbrqoxSQNWACZax9q1a+UefLIn0zf0cZNgpY7+j1ZBv6AsY4tPWxeC8MsDsAF/+qISphX4xlgLn02vF/K3ME+6scXr1q0TKQdJ56qrrpKHY2aCWHLfbbfdBLD4nMk+++wj6fgOQXhrAyyI9gCYMkz9hkIxm8/10kGgcePGya++RM0PwEEAKJ2tV69eKf464XehJGPkHwNlcwy7YXWontkQS/UqFTLoyp0ee+yxtI+A9Prqq6+mva4X6EeNGjXS04y/cb5ngB2DH2LQQVHpiTQ891UKWrBgQQpgqbGdzeZxhISPjZWJMwSh9evXy15PQHPUqFGOCR0Ng/q41+WXXx5XnaSxv5N2A7Y6iaFtDB48WD5rnbbgZlxAVb7iiitkpTmsRv2x2Pqlk0J4vZDHBQUsXR2BqXQUgItZAelJvYKXL1/umjVrJi8B9wMAhhfKL1LVmWeemYLioL7OlMoYvc8555xTwbmN/ADi/vvvL9lRByAFQDnJ4h9Skc7MlWXHEKsvNS7vjBkzsrKHhXVkGuxx9yi3tIsvvjirjwFmK11l8/z0MyjOKE5/QXXD7olZI/TiR1vo3LlzLNB9//338o18NobTr0PSfopJA7AKSe1dYVr0GO0D1wulSZMmZS1taplcfhmrOOHWqVMnpRj8YkxFt2qlZCrQSUEBCzDiwfDhgPSFqc/MRx99JCDGjMksg0QG8a2l3r17yzH/YBCDF+c+7F04ByJV0eEQSXVLSfRrBHQeXB8wYlIWqUs7xu67756snwPsWiqVpVzYeILhHnE+G+IjeSHYRMtU9awUvX8pnucTiLJ9Pu0Dukk8LEcwDgjgAbBYQMK/So3tKsmHZbBP4SIB2OEDp0Tf4R6ocxAuFEr6GSC+BBsSqhiTNv27W7du0jcZT6iOTPxLliyRX8YJLj7sVKCfY8tlzDFpYibRCZ4FL8YM3/PX/gcIYy9mbIS2W7QadjUwPqOgSxsZQwgccRJs+AyFOC6YWwNMgJGodqrnAjYgttqreCkQnfWBBx5IIjlMZCkawEF94yNxkO5lQzVDSoFh5OHlodJFZy1EfpiLbQpPd+pU1VBVRTog4riey41i/uGDgs0rm79oO2Kqs6Q8c4BBDOneRo4ViFSCAlAYtOwCgOiX9JtMkjNGcGw42Fzp02psjzMP0CexudIGdhfwh1mDr2AQB1IXleizEPkAOOoP7WGYArB/AVSMGdpHuwEYgAIbEiYQVWkBSPYzkhe7L/elneTlO2hI56h3jB+kRojVUbzyeX4WlOAf7cJmB9BhMuFZ4tTvVatWJTUUyuAci0NpVVDBJCweCgoRGgBr165dEpm1I/FyMMBjS8DP47777hOGofoxU+kH5XCDgPjyJfo7ujyzF6AU3kcy+X+q69MB+FYUPklsDAZ8MP5jMyEP9jQFMi1rv+XDASY+Bj5EX8I2ymBViZh3jT2Jc/oKf4ALK8h87hlAARAoF0fkQepgIqX+EFw0P8AAaEBMviGxYEPd9FEAAncbJCpAhW/oY5/Uvorpgf6NSwZSOjYkfjG4s0qu9jbGl371FWkIWxmqJloCeRhbgBCaC2CCWodECeiyEIWEiDM1TqD0f7QVNssDgrgQ6QJIaCvmmZC6qFO1JjQTVFVsbEwIBZe6/E2KRh6dE34WrHB/D0wJb6CskE5+z+wK6ZkSvCic8EytkIX7cs3IOOA3iCc8iKRlhDd2J/wSfsLbSBNe8kh4MEibN5sL3sUn4Qd9bFbvW5i44YYbKlzzG9UTXhKSdL+6mfCqXcKDrJxTxk/yckzdXKP+F154IeFXOCWdMeB3eiS8DTfhNRY59sCYrINMPCNlIP/9s2RZSdj4z/udJbyrSJiU8MCY8JNASlqhTgqmEoYzTLpj0DjOExu0j9txT/5w1SVdvWE6M5uqpGE69003o4b57HjL58B5550nn5RGTYojpBakKlSydMb2uHLp0lAz1X4WzUMf15VhpBf9igZmDxaKsFGRVt9rH0g7ELYt3HY8SIh2gn8U9eNgrQ6tuGhwX6Q9VitREZHQMF8goeLigURWr149kTZx8MbAH7rgYJ7BhIINOSRUVG1zmF6I4228qnRlISq2Oo0D5cIBVChUM1Q6Vr7inDf5dAy2IFTLQm7xom7sTIAK4IQaibqIDQ1AwiaMEzbgysIAqhiOyvhjYXLhVwOF4CIEIKHmYfBnJZb8GOUxu2AHQ7VkNZ4JHODjD5UPoQFbVjtvwuEaKinpxf5aQ7X4vEy5DBxrZ3E5ACDg66R2omhrkGiydR6Ols3lHMmKBaJw5ZTzUIrRcyQnwATfPvKrDQk7Fd8mA8ywgUWfifJQWCfSE/Y96uGXMqqdYPuK1pHLM+Urb8GM7vlqoNVTkQN0aAzAqAYsOtDRIDoaHvtGm8YBBnumQVkVYEXLUUFDsCItBJbwnIUj1LuoaQV3HpxSo+mUhaL1kQY46X2RwkLKxJcwX6GPDbAKzeE8168rPKwM0amYCVk9gqIOfnm+tVVXYhxAZWM/IxIU/UBBBlsUrhuojPSXLalfmEpYYp0wU3Pw28Gbnw7J7gGWllmSZ4atbCMqjoPMxkhjLKPjP7PWezKrb1Cm+27KNXygUFVoc+iUuCl1WRnjgHKgqKuE2gj7zY4DOAQCVnyYD69+RHj8efD7QTXMRKyI4oBIeQgDKv422CYKQRiPmeWZ4Y2MA/nigAFWvjhZBfUgsbB6E26EVqOoGlHTNQP7DCqEbjlhJQyDbCFtE3yfzBxy070RS98UDpgNa1O4VoQy6ifDEnS455HN3Nguws3cqHsscbOqxXI80hgxBlnGZpvSe++9J17WLOEjobFPjqVytnWwXM6nT/DXoY64PWosjUNsZ2LpnW0neHBDqJ7e+VCWz/EOZ++bkXEgXxwwCStfnCxwPaz24EzL0rsSe70ADT5VoiDGCiIfJ2RrBo6BN954oxyzyZZVIwy07IPjY4XYs5Cw+B4UWzTw1UEiYisHAJlujxr35ysGABp1Pvjgg9IkJCo+BQTY0Q6ALJ2DpD6D/RoHcuGASVi5cKvIeXHaI5Yg345irxcOgWxgZaVICedGwKtDhw6ShDMkIIXXs25kBcwwuKt6yKZY8rNvUzfU4o+Ubo8aEhSSHd+mYjM4gS5YpeIbUNjFateuLZ+/Vi9rbZv9Ggc2lwMGWJvLwSosDwAgAaF2oerF2Z/YhBtunQCsINQzdtVDSFtIWbr9Ce9mwBBimwbXMNIDcng6A4D6MUbURXb041GNRIXfF/dA2iMfLhZIbov9JmHAy8g4kE8OmEqYT25WQV04/AEocWClt9drfIEAiYk/VgUBFmxV7A9jywfARTp75NiVj5SEeqd2p3R71KgPuxhSGW4L7NbHoM8XEiA+B0S92Mj0Ey/aNvs1DmwOB2wv4eZwrwTLAkqogXyKByDio24c85VMPLVRC/moG7Yv9s8hUfGZE1YNUSf5DhJSHMCWbo8aEhTXFAwBP9L4jA/AxffO8PlCCsPGZWQcyBcHzHE0X5wsoXqQlHB3UEmLpuneM44BF4z3qHCq3vHNp3AbR7o9apSHMMqzOkkdSvhdYcBnsy7Hus1Dr9uvcWBzOWAq4eZysATLAyQhWNHEcO8YYAbQYFhnpY9P9oZgRf5wj5puqCVdifwhWJEOQOmH6AyslFP2m08OGGDlk5tlVheGeJxQUf9Ciu5RC6/ZsXGgmBwwlbCY3Ld7GweMAzlxwCSsnNhlmY0DxoFicsAAq5jct3sbB4wDOXHAACsndllm44BxoJgc+D/zxCIK9zZ0ZQAAAABJRU5ErkJggg=="}},"execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"## importing packages\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\n\nfrom bokeh.layouts import row, column\nfrom bokeh.models import ColumnDataSource, CustomJS, Label, Range1d, Slider, Span\nfrom bokeh.plotting import figure, output_notebook, show\n\noutput_notebook()\n\n## reading data\ndf_train = pd.read_csv(\"/kaggle/input/osic-pulmonary-fibrosis-progression/train.csv\")\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## FVC Distribution\nLet's look at the distribution of the target variable **FVC**.\n","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"sns.distplot(df_train.FVC, color = \"brown\");","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"FVC ranges from **827 to 6399** and seems to have a (visually) normal distribution with **mean 2690** and **standard deviation 833**.\n\n## Evaluation Metric\nNote that not only do we need to predict the FVC but also a confidence value of the prediction. The term *confidence* is a bit confusing (as lower value means more confident) and it might be better to just consider it as the *standard deviation* (or the *uncertainty*).   \n\nLet's define a function and then use it with some examples and understand the metric.\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"## evaluation metric function\ndef laplace_log_likelihood(actual_fvc, predicted_fvc, confidence, return_values = False):\n    \"\"\"\n    Calculates the modified Laplace Log Likelihood score for this competition.\n    \"\"\"\n    sd_clipped = np.maximum(confidence, 70)\n    delta = np.minimum(np.abs(actual_fvc - predicted_fvc), 1000)\n    metric = - np.sqrt(2) * delta / sd_clipped - np.log(np.sqrt(2) * sd_clipped)\n\n    if return_values:\n        return metric\n    else:\n        return np.mean(metric)\n\n\n## default benchmark\nlaplace_log_likelihood(df_train.FVC, np.mean(df_train.FVC), np.std(df_train.FVC))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**-8.023** is the default score to beat while cross-validating models on train data. Any model scoring worse than this is not useful. You can get this default score to beat for each (fold of) validation data as well.\n\n## Constant Prediction\nLet's try out a few combinations of predicting a constant value for FVC as well as confidence to compare how the errors vary with the actual FVC values.\n","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def plot_metric_constants(actual_fvc, constant_fvc, constant_confidence):\n    \"\"\"\n    Generatings a bokeh plot for a constant value of predicted FVC and confidence.\n    \"\"\"\n    lll = laplace_log_likelihood(actual_fvc, constant_fvc, constant_confidence)\n\n    df = pd.DataFrame({\n        \"actual_FVC\": actual_fvc,\n        \"predicted_FVC\": constant_fvc,\n        \"confidence\": constant_confidence,\n        \"metric\": laplace_log_likelihood(actual_fvc, constant_fvc, constant_confidence, return_values = True)\n    }).sort_values(\"actual_FVC\")\n    \n    source = ColumnDataSource(df)\n    \n    tooltips = [\n        (\"Actual FVC\", \"@actual_FVC{0}\"),\n        (\"Predicted FVC\", \"@predicted_FVC{0}\"),\n        (\"Confidence\", \"@confidence{0}\"),\n        (\"Metric\", \"@metric{0.000}\")\n    ]\n    \n    v = figure(\n        plot_width = 345,\n        plot_height = 345,\n        y_range = Range1d(-4, -25),\n        tooltips = tooltips,\n        title = f\"Metric values for FVC = {constant_fvc}, Confidence = {constant_confidence}\"\n    )\n\n    v.circle(\"actual_FVC\", \"metric\", source = source, size = 3, color = \"deepskyblue\", alpha = 0.8)\n    \n    mean = Span(\n        location = lll,\n        dimension = \"width\",\n        line_color = \"red\",\n        line_dash = \"dashed\",\n        line_width = 1.5\n    )\n\n    v.add_layout(mean)\n    \n    score = Label(\n        x = 3500,\n        y = lll + 1.25,\n        text = f\"Laplace Log Likelihood = {round(lll, 3)}\",\n        text_font_size = \"7pt\"\n    )\n\n    v.add_layout(score)\n    \n    v.xaxis.axis_label = \"Actual FVC\"\n    v.yaxis.axis_label = \"Metric Value\"\n\n    return v\n\n\nv1 = plot_metric_constants(df_train.FVC, 2690, 100)\nv2 = plot_metric_constants(df_train.FVC, 2000, 100)\nv3 = plot_metric_constants(df_train.FVC, 3000, 100)\nv4 = plot_metric_constants(df_train.FVC, 4000, 100)\n\nv5 = plot_metric_constants(df_train.FVC, 2000, 833)\nv6 = plot_metric_constants(df_train.FVC, 2690, 833)\nv7 = plot_metric_constants(df_train.FVC, 3000, 833)\nv8 = plot_metric_constants(df_train.FVC, 4000, 833)\n\nv9 = plot_metric_constants(df_train.FVC, 2690, 70)\nv10 = plot_metric_constants(df_train.FVC, 2690, 100)\nv11 = plot_metric_constants(df_train.FVC, 2690, 833)\nv12 = plot_metric_constants(df_train.FVC, 2690, 1000)\n\nshow(column(row(v1, v2), row(v3, v4), row(v5, v6), row(v7, v8), row(v9, v10), row(v11, v12)))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Interactive Simulation\nYou can play around with combinations of FVC values to see how the metric value varies over confidence.\n\n* Any difference between the FVC values greater than 1000 will yield the same metric values since the delta is capped at 1000. \n* The metric value is only dependent on the difference of FVC values and not on the absolute values.\n","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"source = ColumnDataSource(data = dict(\n    x = range(70, 2001),\n    y = [- np.sqrt(2) * 500 / x - np.log(np.sqrt(2) * x) for x in range(70, 2001)]\n))\n\ntooltips = [\n    (\"Confidence\", \"@x\"),\n    (\"Metric\", \"@y\")\n]\n\nv1 = figure(\n    plot_width = 300,\n    plot_height = 300,\n    tooltips = tooltips,\n    title = \"Metric values across confidence\"\n)\n\nv1.line(\"x\", \"y\", source = source, width = 4, color = \"coral\", alpha = 0.8)\n\nv1.y_range.flipped = True\n\nv1.xaxis.axis_label = \"Confidence (Uncertainty)\"\nv1.yaxis.axis_label = \"Metric Value\"\n\nv2 = figure(\n    plot_width = 300,\n    plot_height = 300,\n    tooltips = tooltips,\n    y_range = Range1d(-4, -25),\n    title = \"Metric values across confidence\"\n)\n\nv2.line(\"x\", \"y\", source = source, width = 4, color = \"coral\", alpha = 0.8)\n\nv2.xaxis.axis_label = \"Confidence (Uncertainty)\"\nv2.yaxis.axis_label = \"Metric Value\"\n\n\nslider_actual_FVC = Slider(start = 827, end = 6399, value = 2500, step = 1, title = \"Actual FVC\")\nslider_predicted_FVC = Slider(start = 827, end = 6399, value = 3000, step = 1, title = \"Predicted FVC\")\n\ncallback = CustomJS(args = dict(\n    source = source,\n    actual_FVC = slider_actual_FVC,\n    predicted_FVC = slider_predicted_FVC\n), code = \"\"\"\n    var data = source.data\n    var actual_FVC = actual_FVC.value\n    var predicted_FVC = predicted_FVC.value\n\n    var x = data['x']\n    var y = data['y']\n    var delta = Math.min(Math.abs(actual_FVC - predicted_FVC), 1000)\n\n    for (var i = 0; i < x.length; i++) {\n        y[i] = - Math.sqrt(2) * delta / x[i] - Math.log(Math.sqrt(2) * x[i])\n    }\n\n    source.change.emit();\n\"\"\")\n\nslider_actual_FVC.js_on_change(\"value\", callback)\nslider_predicted_FVC.js_on_change(\"value\", callback)\n\nshow(column(slider_actual_FVC, slider_predicted_FVC, row(v1, v2)))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"While ML models optimize on the point predictions of a target variable (like FVC), it will be interesting to see how the confidence predictions can be optimized. I'm sure through the course of the competition we will see some interesting approaches for the same.\n\n## ML Prediction\nLet's look at the out-of-fold predictions of some of the ML models. The example used below are the predictions from this kernel: https://www.kaggle.com/yasufuminakama/osic-lgb-baseline\n","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"df_preds = pd.read_csv(\"/kaggle/input/osic-lgb-baseline-predictions/train.csv\")\n\ndef plot_fvc_metric_model(actual_fvc, predicted_fvc, predicted_confidence, constant = False):\n    \"\"\"\n    Generatings a bokeh plot for the metric values across actual values of FVC.\n    \"\"\"\n    lll = laplace_log_likelihood(actual_fvc, predicted_fvc, predicted_confidence)\n    metric_values = laplace_log_likelihood(actual_fvc, predicted_fvc, predicted_confidence, return_values = True)\n\n    df = pd.DataFrame({\n        \"actual_FVC\": actual_fvc,\n        \"predicted_FVC\": predicted_fvc,\n        \"confidence\": predicted_confidence,\n        \"metric\": metric_values\n    }).sort_values(\"actual_FVC\")\n    \n    source = ColumnDataSource(df)\n    \n    tooltips = [\n        (\"Actual FVC\", \"@actual_FVC{0}\"),\n        (\"Predicted FVC\", \"@predicted_FVC{0}\"),\n        (\"Confidence\", \"@confidence{0}\"),\n        (\"Metric\", \"@metric{0.000}\")\n    ]\n    \n    if type(predicted_confidence) != int:\n        title = f\"Metric values over FVC\"\n        offset = 0.35\n    else:\n        title = f\"Metric values over FVC (Confidence = {predicted_confidence})\"\n        offset = 0.11\n    \n    v = figure(\n        plot_width = 600,\n        plot_height = 600,\n        y_range = Range1d(max(metric_values) + 0.5, min(metric_values) - 0.5),\n        tooltips = tooltips,\n        title = title\n    )\n\n    v.circle(\"actual_FVC\", \"metric\", source = source, size = 3, color = \"mediumseagreen\", alpha = 0.8)\n    \n    mean = Span(\n        location = lll,\n        dimension = \"width\",\n        line_color = \"red\",\n        line_dash = \"dashed\",\n        line_width = 1.5\n    )\n\n    v.add_layout(mean)\n    \n    score = Label(\n        x = 3750,\n        y = lll + offset,\n        text = f\"Laplace Log Likelihood = {round(lll, 3)}\",\n        text_font_size = \"12pt\"\n    )\n\n    v.add_layout(score)\n    \n    v.xaxis.axis_label = \"Actual FVC\"\n    v.yaxis.axis_label = \"Metric Value\"\n\n    return v\n\n\ndef plot_confidence_metric_model(actual_fvc, predicted_fvc, predicted_confidence):\n    \"\"\"\n    Generatings a bokeh plot for the metric values across predicted values of confidence.\n    \"\"\"\n    lll = laplace_log_likelihood(actual_fvc, predicted_fvc, predicted_confidence)\n    metric_values = laplace_log_likelihood(actual_fvc, predicted_fvc, predicted_confidence, return_values = True)\n\n    df = pd.DataFrame({\n        \"actual_FVC\": actual_fvc,\n        \"predicted_FVC\": predicted_fvc,\n        \"confidence\": predicted_confidence,\n        \"metric\": metric_values\n    }).sort_values(\"confidence\")\n    \n    source = ColumnDataSource(df)\n    \n    tooltips = [\n        (\"Actual FVC\", \"@actual_FVC{0}\"),\n        (\"Predicted FVC\", \"@predicted_FVC{0}\"),\n        (\"Confidence\", \"@confidence{0}\"),\n        (\"Metric\", \"@metric{0.000}\")\n    ]\n    \n    v = figure(\n        plot_width = 600,\n        plot_height = 600,\n        y_range = Range1d(max(metric_values) + 0.5, min(metric_values) - 0.5),\n        tooltips = tooltips,\n        title = \"Metric values over Confidence\"\n    )\n\n    v.circle(\"confidence\", \"metric\", source = source, size = 3, color = \"gold\", alpha = 0.8)\n    \n    mean = Span(\n        location = lll,\n        dimension = \"width\",\n        line_color = \"red\",\n        line_dash = \"dashed\",\n        line_width = 1.5\n    )\n\n    v.add_layout(mean)\n    \n    score = Label(\n        x = 600,\n        y = lll + 0.35,\n        text = f\"Laplace Log Likelihood = {round(lll, 3)}\",\n        text_font_size = \"12pt\"\n    )\n\n    v.add_layout(score)\n    \n    v.xaxis.axis_label = \"Predicted Confidence\"\n    v.yaxis.axis_label = \"Metric Value\"\n\n    return v\n\n\ndef plot_pred_metric_model(actual_fvc, predicted_fvc, predicted_confidence):\n    \"\"\"\n    Generatings a bokeh plot for the metric values across predicted values of FVC.\n    \"\"\"\n    lll = laplace_log_likelihood(actual_fvc, predicted_fvc, predicted_confidence)\n    metric_values = laplace_log_likelihood(actual_fvc, predicted_fvc, predicted_confidence, return_values = True)\n\n    df = pd.DataFrame({\n        \"actual_FVC\": actual_fvc,\n        \"predicted_FVC\": predicted_fvc,\n        \"confidence\": predicted_confidence,\n        \"metric\": metric_values\n    }).sort_values(\"predicted_FVC\")\n    \n    source = ColumnDataSource(df)\n    \n    tooltips = [\n        (\"Actual FVC\", \"@actual_FVC{0}\"),\n        (\"Predicted FVC\", \"@predicted_FVC{0}\"),\n        (\"Confidence\", \"@confidence{0}\"),\n        (\"Metric\", \"@metric{0.000}\")\n    ]\n    \n    v = figure(\n        plot_width = 600,\n        plot_height = 600,\n        y_range = Range1d(max(metric_values) + 0.5, min(metric_values) - 0.5),\n        tooltips = tooltips,\n        title = \"Metric values over predicted FVC\"\n    )\n\n    v.circle(\"predicted_FVC\", \"metric\", source = source, size = 3, color = \"turquoise\", alpha = 0.8)\n    \n    mean = Span(\n        location = lll,\n        dimension = \"width\",\n        line_color = \"red\",\n        line_dash = \"dashed\",\n        line_width = 1.5\n    )\n\n    v.add_layout(mean)\n    \n    score = Label(\n        x = 3500,\n        y = lll + 0.35,\n        text = f\"Laplace Log Likelihood = {round(lll, 3)}\",\n        text_font_size = \"12pt\"\n    )\n\n    v.add_layout(score)\n    \n    v.xaxis.axis_label = \"Predicted FVC\"\n    v.yaxis.axis_label = \"Metric Value\"\n\n    return v\n\n\nsns.pairplot(df_preds[[\"FVC\", \"FVC_pred\", \"Confidence\"]]);\n","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"v1 = plot_fvc_metric_model(df_preds.FVC, df_preds.FVC_pred, df_preds.Confidence)\nv2 = plot_fvc_metric_model(df_preds.FVC, df_preds.FVC_pred, 833)\n\nshow(column(v1, v2))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Optimizing the confidence predictions through a model or optimizer scores **-6.851** like what was done in this model / submission which is better than if just the standard deviation was used to predict confidence which scores **-7.373**. There are some interesting approaching shared in public baseline notebooks of how confidence predictions can be optimized.\n","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"v1 = plot_confidence_metric_model(df_preds.FVC, df_preds.FVC_pred, df_preds.Confidence)\nv2 = plot_pred_metric_model(df_preds.FVC, df_preds.FVC_pred, df_preds.Confidence)\n\nshow(column(v1, v2))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"You can clearly see that if you predict a value with high error and low standard deviation it can lead to large penalties. On the flip side if you predict values with high standard deviation the range of penalties is higher. So that's what you need to balance while optimizing the predictions!\n","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}