{"cells":[{"metadata":{},"cell_type":"markdown","source":" # Background\n \nThis is a type of postmortem on the scoring of the competition. The Laplace Log Likelihood (LLL) metric is non-trivial and not at all intuitive. \n\nI probably should have done this review at the beginning, to try and gain some insight and set my expectations straight. So forgive me for getting to this only now, but hopefully the community will be able to benefit from this in future attempts of creating models that are meaningfully better than the ones the competition managed to yield.\n\nThe winning kernel of the competition was this one: https://www.kaggle.com/artkulak/inference-45-55-600-epochs-tuned-effnet-b5-30-ep\n\nAs many other submitted kernels, it was based on an ensemble of two approaches:\n\n1. Linear Decay (based on EfficientNets) - trying to predict the linear coefficient of the decay in FVC by week, using also CT image data.\n1. Multiple-Quantile Regression - trying to predict the FVC by week directly, based solely on the provided tabular data, and using the quantiles to assess the confidence of the prediction.\n\nI will be focusing here on the first approach, since it seems that there was no true benefit from using the CT images in this competition, and I find it hard to believe that there are no meaningful features to be extracted from them. \nI will try to figure out basically why the Linear Decay approach failed so far, what we were meant to expect in the first place, and what can be done moving on."},{"metadata":{},"cell_type":"markdown","source":"# Imports & setup"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import itertools\nimport pandas as pd\nimport numpy as np \nimport matplotlib.pyplot as plt \nfrom sklearn.metrics import *\nimport seaborn as sns\n\nfrom ipywidgets import interact, interactive, fixed, interact_manual\nimport ipywidgets as widgets","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def add_one_to_one_correlation_line(ax, min_factor=1, max_factor=1, **plot_kwargs):\n    lim_min, lim_max = pd.DataFrame([ax.get_ylim(), ax.get_xlim()]).agg({0: 'min', 1: 'max'})\n    lim_min *= min_factor\n    lim_max *= max_factor\n    plot_kwargs_internal = dict(color='grey', ls='--')\n    plot_kwargs_internal.update(plot_kwargs)\n    ax.plot([lim_min, lim_max], [lim_min, lim_max], **plot_kwargs_internal)\n    ax.set_ylim([lim_min, lim_max])\n    ax.set_xlim([lim_min, lim_max])\n    \ndef laplace_log_likelihood(actual_fvc, predicted_fvc, confidence, return_values=False):\n    \"\"\"\n    Calculates the modified Laplace Log Likelihood score for this competition.\n    \"\"\"\n    sd_clipped = np.maximum(confidence, 70)\n    delta = np.minimum(np.abs(actual_fvc - predicted_fvc), 1000)\n    metric = - np.sqrt(2) * delta / sd_clipped - np.log(np.sqrt(2) * sd_clipped)\n\n    if return_values:\n        return metric\n    else:\n        return np.mean(metric)\n    \nplt.rcParams.update({\n    'font.size': 14,\n    'figure.facecolor': 'white',\n    'errorbar.capsize': 4,\n    'figure.figsize': (9, 6),\n    'figure.dpi': 72.0,\n    'grid.color': 'k',\n    'grid.linestyle': '-',\n    'grid.linewidth': 0.5,\n})\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Our basic train data"},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/train.csv') ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load winning model's train data prediction"},{"metadata":{},"cell_type":"markdown","source":"The following dictionary contains predictions of the linear decay coefficient \"a\", predicted on the train data using the part of the **winning model** that was based on EfficientNet B5, and trained on the CT images.\n\nI copied the data here so as to not need to run the prediction each time."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"df_a_dict = {\n    'a_true': {'ID00007637202177411956430': -3.1671258700696305, 'ID00009637202177434476278': -9.379955123410653, 'ID00010637202177584971671': -17.042803238739005, 'ID00011637202177653955184': -4.548925129725728, 'ID00012637202177665765362': -8.543078528986666, 'ID00014637202177757139317': -14.033411631297636, 'ID00015637202177877247924': -7.5789593481130995, 'ID00019637202178323708467': -5.72996072996071, 'ID00020637202178344345685': -7.370692539310795, 'ID00023637202179104603099': -2.549151772190283, 'ID00025637202179541264076': -10.818152815441035, 'ID00026637202179561894768': -2.9678593086719522, 'ID00027637202179689871102': -5.494368559789142, 'ID00030637202181211009029': -4.687772738940671, 'ID00032637202181710233084': -14.36286855036857, 'ID00035637202182204917484': -9.673158832184663, 'ID00038637202182690843176': -0.8007971241012766, 'ID00042637202184406822975': 0.5981338481338266, 'ID00047637202184938901501': -10.83154246100512, 'ID00048637202185016727717': -1.1904083570750204, 'ID00051637202185848464638': -3.3735904813748943, 'ID00052637202186188008618': -23.287500000000005, 'ID00060637202187965290703': -6.74182278869783, 'ID00061637202188184085559': 8.241642441860474, 'ID00062637202188654068490': -3.7984441301273133, 'ID00067637202189903532242': -5.9531843104176225, 'ID00068637202190879923934': -6.6029635258358494, 'ID00072637202198161894406': -2.0348133198789276, 'ID00073637202198167792918': -18.38177852681676, 'ID00075637202198610425520': -8.13924528301889, 'ID00076637202199015035026': 11.225947841332374, 'ID00077637202199102000916': 4.1800711743772645, 'ID00078637202199415319443': -7.029780085522318, 'ID00082637202201836229724': -0.06711804533412816, 'ID00086637202203494931510': -0.19162534050429111, 'ID00089637202204675567570': -12.218611813013307, 'ID00090637202204766623410': -5.73331843575418, 'ID00093637202205278167493': 0.7165697674418419, 'ID00094637202205333947361': -2.837139561707097, 'ID00099637202206203080121': -3.62201921936174, 'ID00102637202206574119190': -0.43741258741261607, 'ID00104637202208063407045': -16.464817048652975, 'ID00105637202208831864134': -5.873595306252803, 'ID00108637202209619669361': -4.185046281519326, 'ID00109637202210454292264': -6.038142355569738, 'ID00110637202210673668310': -28.182574916759165, 'ID00111637202210956877205': -11.658906334596628, 'ID00115637202211874187958': -6.403378656777947, 'ID00117637202212360228007': 1.988795518207333, 'ID00119637202215426335765': 0.6563821493539476, 'ID00122637202216437668965': -7.18235252710125, 'ID00123637202217151272140': -3.6462542252967487, 'ID00124637202217596410344': -7.779256408156603, 'ID00125637202218590429387': -15.24165142912136, 'ID00126637202218610655908': 1.8965999411245564, 'ID00127637202219096738943': -4.2896274937655905, 'ID00128637202219474716089': -8.912794398472291, 'ID00129637202219868188000': -2.335302293259194, 'ID00130637202220059448013': -8.003377003824559, 'ID00131637202220424084844': -5.947910907511261, 'ID00132637202222178761324': -0.6589328033025296, 'ID00133637202223847701934': -2.9042331714087317, 'ID00134637202223873059688': -9.75655313137658, 'ID00135637202224630271439': -0.5983642154024482, 'ID00136637202224951350618': -1.1424745789024495, 'ID00138637202231603868088': -4.240429505135428, 'ID00139637202231703564336': -3.944482058226148, 'ID00140637202231728595149': -1.510682768230298, 'ID00149637202232704462834': -8.940533230293632, 'ID00161637202235731948764': -0.568472381576776, 'ID00165637202237320314458': 1.4750060081711185, 'ID00167637202237397919352': -11.627400164919225, 'ID00168637202237852027833': -8.515749296906424, 'ID00169637202238024117706': -3.0797419040662377, 'ID00170637202238079193844': 4.677616413093594, 'ID00172637202238316925179': -0.5767410412440729, 'ID00173637202238329754031': -4.263502331879073, 'ID00180637202240177410333': -4.471508470454703, 'ID00183637202241995351650': 0.4055631591659285, 'ID00184637202242062969203': -4.070408875785584, 'ID00186637202242472088675': -16.12174590802803, 'ID00190637202244450116191': -1.3071039079066564, 'ID00192637202245493238298': -4.208603606814463, 'ID00196637202246668775836': -1.0805557636132173, 'ID00197637202246865691526': 14.68261173184358, 'ID00199637202248141386743': 1.8671517002973872, 'ID00202637202249376026949': -12.683186241231025, 'ID00207637202252526380974': -3.506461838575942, 'ID00210637202257228694086': 0.6937716262975414, 'ID00213637202257692916109': -1.4488881778915732, 'ID00214637202257820847190': -1.5641642472003332, 'ID00216637202257988213445': -3.3800276265540297, 'ID00218637202258156844710': 3.9701631701631857, 'ID00219637202258203123958': -3.8737373737373324, 'ID00221637202258717315571': -1.850290697674443, 'ID00222637202259066229764': -4.212131519274413, 'ID00224637202259281193413': 0.320381462799496, 'ID00225637202259339837603': -7.460478801658403, 'ID00228637202259965313869': -5.131120604666424, 'ID00229637202260254240583': -3.410314314683338, 'ID00232637202260377586117': -1.408633897223125, 'ID00233637202260580149633': -4.185695934713664, 'ID00234637202261078001846': -2.2809869375907277, 'ID00235637202261451839085': 2.5571496891439374, 'ID00240637202264138860065': -4.816082934082311, 'ID00241637202264294508775': -3.130466007022046, 'ID00242637202264759739921': -2.5840534171248914, 'ID00248637202266698862378': 1.358889516584587, 'ID00249637202266730854017': -24.291011284911363, 'ID00251637202267455595113': -5.585896631094386, 'ID00255637202267923028520': -2.31592039800996, 'ID00264637202270643353440': 0.26533802266507095, 'ID00267637202270790561585': -6.5318940137389605, 'ID00273637202271319294586': 1.4789915966386689, 'ID00275637202271440119890': -10.103654644111094, 'ID00276637202271694539978': -10.778018575851366, 'ID00279637202272164826258': -5.1659110451027805, 'ID00283637202278714365037': 0.6811119203689997, 'ID00285637202278913507108': 0.3500638365783564, 'ID00288637202279148973731': 9.052208392994826, 'ID00290637202279304677843': -9.107582307948078, 'ID00291637202279398396106': -9.081413151649674, 'ID00294637202279614924243': -4.406274479085077, 'ID00296637202279895784347': -0.24566929133860993, 'ID00298637202280361773446': -8.115703486954404, 'ID00299637202280383305867': -2.356115107913677, 'ID00305637202281772703145': -4.808091286307084, 'ID00307637202282126172865': -8.931895831751602, 'ID00309637202282195513787': -4.441007194244559, 'ID00312637202282607344793': -4.980366889023662, 'ID00317637202283194142136': -10.93991769547329, 'ID00319637202283897208687': 7.197137650899697, 'ID00322637202284842245491': -6.314690529455673, 'ID00323637202285211956970': 4.51550035342825, 'ID00329637202285906759848': -1.040237518556106, 'ID00331637202286306023714': 2.561119350558103, 'ID00335637202286784464927': -5.53088662790696, 'ID00336637202286801879145': -6.041440015824283, 'ID00337637202286839091062': -0.03774879890182057, 'ID00339637202287377736231': -9.381847725162507, 'ID00340637202287399835821': -3.8140480591497377, 'ID00341637202287410878488': -2.016677130748708, 'ID00342637202287526592911': -8.224489014774575, 'ID00343637202287577133798': -0.28084152275546265, 'ID00344637202287684217717': -8.564457557875643, 'ID00351637202289476567312': 1.6565740356698286, 'ID00355637202295106567614': 7.3312581913498915, 'ID00358637202295388077032': -0.26516072980018596, 'ID00360637202295712204040': -2.216851387428209, 'ID00364637202296074419422': -15.030662522741034, 'ID00365637202296085035729': -8.189459707895, 'ID00367637202296290303449': -8.287634633046654, 'ID00368637202296470751086': -2.220929351121908, 'ID00370637202296737666151': -0.330568228950645, 'ID00371637202296828615743': -3.6651212553494945, 'ID00376637202297677828573': -3.4435780191067793, 'ID00378637202298597306391': -3.9566950676150774, 'ID00381637202299644114027': -15.329545454545421, 'ID00383637202300493233675': -4.497841726618682, 'ID00388637202301028491611': -5.166130230255007, 'ID00392637202302319160044': -7.61721991701238, 'ID00393637202302431697467': -3.444329183955729, 'ID00398637202303897337979': -23.114528530959117, 'ID00400637202305055099402': -0.9299217109031861, 'ID00401637202305320178010': -0.8837943124269375, 'ID00405637202308359492977': -1.1790502158600187, 'ID00407637202308788732304': -4.5130490133672945, 'ID00408637202308839708961': 0.016659637283834548, 'ID00411637202309374271828': -0.9568034557235356, 'ID00414637202310318891556': -20.453681433780712, 'ID00417637202310901214011': -1.1616005215548653, 'ID00419637202311204720264': -2.1359750738994987, 'ID00421637202311550012437': -1.55402989344205, 'ID00422637202311677017371': -4.740104413858578, 'ID00423637202312137826377': -9.214885618301077, 'ID00426637202313170790466': -1.0813172043010886}, \n    'a_pred': {'ID00007637202177411956430': -3.8612202405929565, 'ID00009637202177434476278': -3.862869143486023, 'ID00010637202177584971671': -3.871055245399475, 'ID00011637202177653955184': np.nan, 'ID00012637202177665765362': -3.5996041297912598, 'ID00014637202177757139317': -3.8569345474243164, 'ID00015637202177877247924': -3.8619956970214844, 'ID00019637202178323708467': -3.8805158138275146, 'ID00020637202178344345685': -3.5990827083587646, 'ID00023637202179104603099': -3.8642048835754395, 'ID00025637202179541264076': -3.9143260717391968, 'ID00026637202179561894768': -3.9275262355804443, 'ID00027637202179689871102': -3.8610132932662964, 'ID00030637202181211009029': -3.871375322341919, 'ID00032637202181710233084': -3.8741509914398193, 'ID00035637202182204917484': -3.8920458555221558, 'ID00038637202182690843176': -3.8615851402282715, 'ID00042637202184406822975': -3.9205071926116943, 'ID00047637202184938901501': -3.8645107746124268, 'ID00048637202185016727717': -3.578603744506836, 'ID00051637202185848464638': -3.861521005630493, 'ID00052637202186188008618': -3.875973701477051, 'ID00060637202187965290703': -3.8684475421905518, 'ID00061637202188184085559': -3.892069101333618, 'ID00062637202188654068490': -3.5683635473251343, 'ID00067637202189903532242': -3.9910826683044434, 'ID00068637202190879923934': -3.8631844520568848, 'ID00072637202198161894406': -3.8902249336242676, 'ID00073637202198167792918': -3.872480630874634, 'ID00075637202198610425520': -3.5864851474761963, 'ID00076637202199015035026': -3.6153374910354614, 'ID00077637202199102000916': -3.8639262914657593, 'ID00078637202199415319443': -3.9357398748397827, 'ID00082637202201836229724': -3.678354501724243, 'ID00086637202203494931510': -3.6851218938827515, 'ID00089637202204675567570': -3.578460216522217, 'ID00090637202204766623410': -3.864747166633606, 'ID00093637202205278167493': -3.864656686782837, 'ID00094637202205333947361': -3.8413201570510864, 'ID00099637202206203080121': -3.8662099838256836, 'ID00102637202206574119190': -3.877243995666504, 'ID00104637202208063407045': -3.600427269935608, 'ID00105637202208831864134': -3.868635416030884, 'ID00108637202209619669361': -3.9092389345169067, 'ID00109637202210454292264': -3.866293430328369, 'ID00110637202210673668310': -3.908149480819702, 'ID00111637202210956877205': -3.8607717752456665, 'ID00115637202211874187958': -3.858007311820984, 'ID00117637202212360228007': -3.6103405952453613, 'ID00119637202215426335765': -3.8734703063964844, 'ID00122637202216437668965': -3.990039348602295, 'ID00123637202217151272140': -3.5734548568725586, 'ID00124637202217596410344': -3.870888829231262, 'ID00125637202218590429387': -3.5813323259353638, 'ID00126637202218610655908': -3.855828285217285, 'ID00127637202219096738943': -3.874549388885498, 'ID00128637202219474716089': -3.6313353776931763, 'ID00129637202219868188000': -3.595435857772827, 'ID00130637202220059448013': -3.5767300128936768, 'ID00131637202220424084844': -3.6112489700317383, 'ID00132637202222178761324': -3.988365650177002, 'ID00133637202223847701934': -3.5620946884155273, 'ID00134637202223873059688': -3.8661530017852783, 'ID00135637202224630271439': -3.606541633605957, 'ID00136637202224951350618': -3.864742636680603, 'ID00138637202231603868088': -3.8659673929214478, 'ID00139637202231703564336': -3.8585243225097656, 'ID00140637202231728595149': -3.6186723709106445, 'ID00149637202232704462834': -3.870980978012085, 'ID00161637202235731948764': -3.8690147399902344, 'ID00165637202237320314458': -3.8802510499954224, 'ID00167637202237397919352': -3.877613067626953, 'ID00168637202237852027833': -3.8709957599639893, 'ID00169637202238024117706': -3.625394344329834, 'ID00170637202238079193844': -3.8573580980300903, 'ID00172637202238316925179': -3.9281067848205566, 'ID00173637202238329754031': -3.861555337905884, 'ID00180637202240177410333': -3.8944649696350098, 'ID00183637202241995351650': -3.9092206954956055, 'ID00184637202242062969203': -3.8779832124710083, 'ID00186637202242472088675': -3.8646456003189087, 'ID00190637202244450116191': -3.868389129638672, 'ID00192637202245493238298': -3.585062265396118, 'ID00196637202246668775836': -3.86798357963562, 'ID00197637202246865691526': -3.614446997642517, 'ID00199637202248141386743': -3.9034790992736816, 'ID00202637202249376026949': -3.604743480682373, 'ID00207637202252526380974': -3.8882341384887695, 'ID00210637202257228694086': -3.633828639984131, 'ID00213637202257692916109': -3.6097298860549927, 'ID00214637202257820847190': -3.9147491455078125, 'ID00216637202257988213445': -3.5819523334503174, 'ID00218637202258156844710': -3.8558605909347534, 'ID00219637202258203123958': -3.861579418182373, 'ID00221637202258717315571': -3.86562716960907, 'ID00222637202259066229764': -3.863606810569763, 'ID00224637202259281193413': -3.5707242488861084, 'ID00225637202259339837603': -3.5665721893310547, 'ID00228637202259965313869': -3.873555898666382, 'ID00229637202260254240583': -3.8874735832214355, 'ID00232637202260377586117': -3.6031644344329834, 'ID00233637202260580149633': -3.865260362625122, 'ID00234637202261078001846': -3.8925886154174805, 'ID00235637202261451839085': -3.8659271001815796, 'ID00240637202264138860065': -3.8419206142425537, 'ID00241637202264294508775': -3.6060891151428223, 'ID00242637202264759739921': -3.917689323425293, 'ID00248637202266698862378': -3.625789999961853, 'ID00249637202266730854017': -3.9051833152770996, 'ID00251637202267455595113': -3.8495161533355713, 'ID00255637202267923028520': -3.5779985189437866, 'ID00264637202270643353440': -3.566640853881836, 'ID00267637202270790561585': -3.5988450050354004, 'ID00273637202271319294586': -3.871098279953003, 'ID00275637202271440119890': -3.60494863986969, 'ID00276637202271694539978': -3.8570327758789062, 'ID00279637202272164826258': -3.868941307067871, 'ID00283637202278714365037': -3.5793484449386597, 'ID00285637202278913507108': -3.575894355773926, 'ID00288637202279148973731': -3.8735222816467285, 'ID00290637202279304677843': -3.5938663482666016, 'ID00291637202279398396106': -3.861234426498413, 'ID00294637202279614924243': -3.8664491176605225, 'ID00296637202279895784347': -3.872938632965088, 'ID00298637202280361773446': -3.6178619861602783, 'ID00299637202280383305867': -3.858301281929016, 'ID00305637202281772703145': -3.8689846992492676, 'ID00307637202282126172865': -3.5746545791625977, 'ID00309637202282195513787': -3.5697460174560547, 'ID00312637202282607344793': -3.909405469894409, 'ID00317637202283194142136': -3.8721213340759277, 'ID00319637202283897208687': -3.8601895570755005, 'ID00322637202284842245491': -3.876878261566162, 'ID00323637202285211956970': -3.8580846786499023, 'ID00329637202285906759848': -3.863345742225647, 'ID00331637202286306023714': -3.6117782592773438, 'ID00335637202286784464927': -3.8927420377731323, 'ID00336637202286801879145': -3.8738198280334473, 'ID00337637202286839091062': -3.864347219467163, 'ID00339637202287377736231': -3.8678271770477295, 'ID00340637202287399835821': -3.8646817207336426, 'ID00341637202287410878488': -3.612298011779785, 'ID00342637202287526592911': -3.865936279296875, 'ID00343637202287577133798': -3.6235780715942383, 'ID00344637202287684217717': -3.6293423175811768, 'ID00351637202289476567312': -3.8576533794403076, 'ID00355637202295106567614': -3.613119125366211, 'ID00358637202295388077032': -3.86765718460083, 'ID00360637202295712204040': -3.5651514530181885, 'ID00364637202296074419422': -3.8689231872558594, 'ID00365637202296085035729': -3.8625491857528687, 'ID00367637202296290303449': -3.5815742015838623, 'ID00368637202296470751086': -3.5748727321624756, 'ID00370637202296737666151': -3.8592185974121094, 'ID00371637202296828615743': -3.864688754081726, 'ID00376637202297677828573': -3.5712382793426514, 'ID00378637202298597306391': -3.582982659339905, 'ID00381637202299644114027': -3.875988721847534, 'ID00383637202300493233675': -3.577014684677124, 'ID00388637202301028491611': -3.875, 'ID00392637202302319160044': -3.8661292791366577, 'ID00393637202302431697467': -3.5734773874282837, 'ID00398637202303897337979': -3.8632748126983643, 'ID00400637202305055099402': -3.873966693878174, 'ID00401637202305320178010': -3.8652416467666626, 'ID00405637202308359492977': -3.5791029930114746, 'ID00407637202308788732304': -3.8651294708251953, 'ID00408637202308839708961': -3.859568953514099, 'ID00411637202309374271828': -3.8668934106826782, 'ID00414637202310318891556': -3.5747859477996826, 'ID00417637202310901214011': -3.574668526649475, 'ID00419637202311204720264': -3.8840116262435913, 'ID00421637202311550012437': -3.8655506372451782, 'ID00422637202311677017371': -3.8667728900909424, 'ID00423637202312137826377': -3.86196231842041, 'ID00426637202313170790466': -3.569868326187134}, \n    'b': {'ID00007637202177411956430': 2299.555119037628, 'ID00009637202177434476278': 3690.902953147888, 'ID00010637202177584971671': 3523.0, 'ID00011637202177653955184': np.nan, 'ID00012637202177665765362': 3536.7869362831116, 'ID00014637202177757139317': 3807.0, 'ID00015637202177877247924': 3088.343948364258, 'ID00019637202178323708467': 2150.4467055797577, 'ID00020637202178344345685': 2361.7834887504578, 'ID00023637202179104603099': 1524.4073853492737, 'ID00025637202179541264076': 2903.0, 'ID00026637202179561894768': 3045.028575658798, 'ID00027637202179689871102': 2518.3321595191956, 'ID00030637202181211009029': 2579.5561311244965, 'ID00032637202181710233084': 5161.224529743195, 'ID00035637202182204917484': 2770.8125044107437, 'ID00038637202182690843176': 3969.1695108413696, 'ID00042637202184406822975': 3794.9205071926117, 'ID00047637202184938901501': 3320.729021549225, 'ID00048637202185016727717': 1407.2074337005615, 'ID00051637202185848464638': 1693.1384789943695, 'ID00052637202186188008618': 2675.0467376708984, 'ID00060637202187965290703': 3316.895160675049, 'ID00061637202188184085559': 4058.517589330673, 'ID00062637202188654068490': 2458.007996082306, 'ID00067637202189903532242': 3471.964330673218, 'ID00068637202190879923934': 2836.4950289726257, 'ID00072637202198161894406': 2417.353374004364, 'ID00073637202198167792918': 2558.617441892624, 'ID00075637202198610425520': 1730.662128686905, 'ID00076637202199015035026': 2283.538650035858, 'ID00077637202199102000916': 3189.319631457329, 'ID00078637202199415319443': 1860.0, 'ID00082637202201836229724': 2987.8887355327606, 'ID00086637202203494931510': 3348.5743905305862, 'ID00089637202204675567570': 2503.0492215156555, 'ID00090637202204766623410': 3627.7007018327713, 'ID00093637202205278167493': 3691.135343313217, 'ID00094637202205333947361': 4916.0, 'ID00099637202206203080121': 3104.11323928833, 'ID00102637202206574119190': 2564.6317319869995, 'ID00104637202208063407045': 2449.008545398712, 'ID00105637202208831864134': 2649.219958305359, 'ID00108637202209619669361': 2957.001628279686, 'ID00109637202210454292264': 2277.9135518074036, 'ID00110637202210673668310': 2053.6222422122955, 'ID00111637202210956877205': 2423.746945977211, 'ID00115637202211874187958': 2605.8701096773148, 'ID00117637202212360228007': 2947.4413623809814, 'ID00119637202215426335765': 2924.746940612793, 'ID00122637202216437668965': 2565.039842605591, 'ID00123637202217151272140': 1845.5258331298828, 'ID00124637202217596410344': 3740.9317749738693, 'ID00125637202218590429387': 2377.650658607483, 'ID00126637202218610655908': 2444.404909133911, 'ID00127637202219096738943': 1847.480173110962, 'ID00128637202219474716089': 2343.465402841568, 'ID00129637202219868188000': 2253.0, 'ID00130637202220059448013': 1729.3440301418304, 'ID00131637202220424084844': 3554.5086851119995, 'ID00132637202222178761324': 2605.930193901062, 'ID00133637202223847701934': 3187.875810623169, 'ID00134637202223873059688': 2636.7323060035706, 'ID00135637202224630271439': 3919.8027906417847, 'ID00136637202224951350618': 3250.594227910042, 'ID00138637202231603868088': 4564.1235435009, 'ID00139637202231703564336': 3854.7363891601562, 'ID00140637202231728595149': 4181.89875793457, 'ID00149637202232704462834': 3417.3231630325317, 'ID00161637202235731948764': 2871.952117919922, 'ID00165637202237320314458': 2408.5398329496384, 'ID00167637202237397919352': 4589.143291473389, 'ID00168637202237852027833': 3213.096970319748, 'ID00169637202238024117706': 2563.504732131958, 'ID00170637202238079193844': 2116.5796077251434, 'ID00172637202238316925179': 2740.4837374687195, 'ID00173637202238329754031': 2690.4771087169647, 'ID00180637202240177410333': 3044.050184726715, 'ID00183637202241995351650': 2693.4596071243286, 'ID00184637202242062969203': 3242.4615131616592, 'ID00186637202242472088675': 3126.9171648025513, 'ID00190637202244450116191': 4306.473556518555, 'ID00192637202245493238298': 1742.3402490615845, 'ID00196637202246668775836': 3883.8118522167206, 'ID00197637202246865691526': 2776.0, 'ID00199637202248141386743': 2691.5173954963684, 'ID00202637202249376026949': 4383.0948696136475, 'ID00207637202252526380974': 2507.3117265701294, 'ID00210637202257228694086': 2846.0, 'ID00213637202257692916109': 3056.65837931633, 'ID00214637202257820847190': 2880.7442474365234, 'ID00216637202257988213445': 2693.819523334503, 'ID00218637202258156844710': 2516.117211818695, 'ID00219637202258203123958': 6399.0, 'ID00221637202258717315571': 1443.5218988656998, 'ID00222637202259066229764': 2640.1363931894302, 'ID00224637202259281193413': 2992.0, 'ID00225637202259339837603': 1629.3654384613037, 'ID00228637202259965313869': 1939.2297825813293, 'ID00229637202260254240583': 3607.411892414093, 'ID00232637202260377586117': 1941.4284799098969, 'ID00233637202260580149633': 3817.4042189121246, 'ID00234637202261078001846': 2091.6777658462524, 'ID00235637202261451839085': 2192.4415214061737, 'ID00240637202264138860065': 3010.2096030712128, 'ID00241637202264294508775': 1406.2121782302856, 'ID00242637202264759739921': 1868.6707572937012, 'ID00248637202266698862378': 1322.761059999466, 'ID00249637202266730854017': 3387.7762327194214, 'ID00251637202267455595113': 2155.839839220047, 'ID00255637202267923028520': 1566.5779985189438, 'ID00264637202270643353440': 1984.3663311004639, 'ID00267637202270790561585': 1025.7965350151062, 'ID00273637202271319294586': 3035.484393119812, 'ID00275637202271440119890': 2253.037534713745, 'ID00276637202271694539978': 3130.1421966552734, 'ID00279637202272164826258': 4171.951530456543, 'ID00283637202278714365037': 1965.0, 'ID00285637202278913507108': 1425.3035774230957, 'ID00288637202279148973731': 2357.3439679145813, 'ID00290637202279304677843': 2297.3754653930664, 'ID00291637202279398396106': 4058.722468852997, 'ID00294637202279614924243': 2346.3322455883026, 'ID00296637202279895784347': 2242.3646931648254, 'ID00298637202280361773446': 1894.471447944641, 'ID00299637202280383305867': 2193.8826282024384, 'ID00305637202281772703145': 3014.951877593994, 'ID00307637202282126172865': 3180.723963737488, 'ID00309637202282195513787': 3250.709238052368, 'ID00312637202282607344793': 1879.0, 'ID00317637202283194142136': 2475.675154685974, 'ID00319637202283897208687': 4189.322274684906, 'ID00322637202284842245491': 2471.6306347846985, 'ID00323637202285211956970': 1512.3102016448975, 'ID00329637202285906759848': 2955.6704839468002, 'ID00331637202286306023714': 3236.1297912597656, 'ID00335637202286784464927': 3054.6782261133194, 'ID00336637202286801879145': 2592.2429189682007, 'ID00337637202286839091062': 4071.0660693645477, 'ID00339637202287377736231': 3574.0, 'ID00340637202287399835821': 2341.1353182792664, 'ID00341637202287410878488': 2950.144344329834, 'ID00342637202287526592911': 2478.916534423828, 'ID00343637202287577133798': 1720.3422956466675, 'ID00344637202287684217717': 2066.0, 'ID00351637202289476567312': 2862.0071473121643, 'ID00355637202295106567614': 4504.714859008789, 'ID00358637202295388077032': 2045.4706287384033, 'ID00360637202295712204040': 3117.6954543590546, 'ID00364637202296074419422': 3334.150157928467, 'ID00365637202296085035729': 2556.1892215013504, 'ID00367637202296290303449': 1491.3550970554352, 'ID00368637202296470751086': 3153.0241091251373, 'ID00370637202296737666151': 2354.3106231689453, 'ID00371637202296828615743': 2493.1881325244904, 'ID00376637202297677828573': 4423.278292894363, 'ID00378637202298597306391': 3427.5745664834976, 'ID00381637202299644114027': 2434.3799436092377, 'ID00383637202300493233675': 3408.696704864502, 'ID00388637202301028491611': 3157.0, 'ID00392637202302319160044': 2250.3306463956833, 'ID00393637202302431697467': 3562.0, 'ID00398637202303897337979': 3088.496022939682, 'ID00400637202305055099402': 4074.345034122467, 'ID00401637202305320178010': 2346.3401491642, 'ID00405637202308359492977': 1560.0, 'ID00407637202308788732304': 3260.167718887329, 'ID00408637202308839708961': 2701.1743963956833, 'ID00411637202309374271828': 1583.0682538747787, 'ID00414637202310318891556': 2644.023501634598, 'ID00417637202310901214011': 3385.597348213196, 'ID00419637202311204720264': 3043.3040697574615, 'ID00421637202311550012437': 2796.9832595586777, 'ID00422637202311677017371': 1953.2006373405457, 'ID00423637202312137826377': 3359.653359413147, 'ID00426637202313170790466': 2925.0}\n}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_a = pd.DataFrame(df_a_dict)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Interesting plots regarding LLL score"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def get_score_df(fix_rate, sigma_fraction, last_three=True):\n    df_a_fixed = df_a.copy()\n    df_a_fixed['pred_fixed'] = df_a_fixed['a_true'] * (fix_rate) + df_a['a_pred'] * (1-fix_rate)\n\n    train_pred = train.copy()\n    df_a_ri = df_a_fixed.reindex(train_pred['Patient']).reset_index()\n    train_pred['pred_fvc'] = df_a_ri['pred_fixed'] * train_pred['Weeks'] + df_a_ri['b']\n    train_pred['pred_confidence'] = sigma_fraction * train_pred['pred_fvc']\n    if last_three:\n        # Keep only the final three visits, as only these get scored on the test set\n        train_pred = train_pred.groupby('Patient').tail(3)\n    return df_a_fixed, train_pred","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The following interactive graph shows the prediction of \"a\" given by the EffNet model for each train set patient.\n\nAs you can clearly see (thanks to the added 1:1 diagonal line), **the prediction was pretty much constant**. What does this tell us? That using the CT images did not have any real benefit.\n\nMoreover, I added a \"fix rate\" parameter, that allows to artificially bring the R^2 of the \"a\" prediction all the way to 1.0. Apparently, even this doesn't get us very far. Bear in mind that the confidence value I used here is a constant fraction * the predicted FVC. In addition, only the last three visits for each patient were considered, as only these get scored on the test set.\n\nPlay around with the interactive graph to get a sense of the LLL score regarding the predicted linear decay."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def get_score_graph(fix_rate, sigma_fraction):\n    df_a, train_pred = get_score_df(fix_rate, sigma_fraction)\n    \n    fig, axes = plt.subplots(1, 2, figsize=(12, 5), dpi=70)\n    ax = axes[0]\n    sns.scatterplot(x='a_true', y='pred_fixed', data=df_a, ax=ax)\n    add_one_to_one_correlation_line(ax)\n    ax.set_title('Predicted \"a\" values')\n    r2 = r2_score(df_a.dropna()['a_true'], df_a.dropna()['pred_fixed'])\n    ax.text(0.05, 0.95, f'$R^2={r2:.2f}$', transform=ax.transAxes, va='top', ha='left')\n\n    ax = axes[1]\n    plt.figure(figsize=(6, 6), dpi=80)\n    sns.scatterplot(x='FVC', y='pred_fvc', data=train_pred, ax=ax)\n    add_one_to_one_correlation_line(ax)\n    ax.set_title('Predicted FVC values by corresponding \"a\"')\n    lll = laplace_log_likelihood(train_pred['FVC'], train_pred['pred_fvc'], train_pred['pred_confidence'])\n    ax.text(0.05, 0.95, f'$LLL={lll:.4f}$\\n$\\\\sigma={sigma_fraction}*FVC$', transform=ax.transAxes, va='top', ha='left')\n\n    title = 'Predicted train set values by winning effnet model'\n    if fix_rate > 0:\n        title += f' - corrected by {fix_rate:.0%}'\n    fig.suptitle(title)\n    plt.tight_layout(rect=[0, 0.03, 1, 0.8])  # The 0.95 part defines the extra space from the top\n    fig.subplots_adjust(wspace=0.25)\n    plt.close('all')\n    return fig\n\ninteract(\n    get_score_graph, \n    fix_rate=widgets.FloatSlider(min=0, max=1, step=0.05, value=0), \n    sigma_fraction=widgets.FloatSlider(min=0.02, max=0.3, step=0.02, value=0.1), \n);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"After playing around with this interactive graph, we can see that even an exact prediction of linear decay has little effect on the final score. \n\nI'm adding here two non-interactive graphs for better documentation: \n1. The exact predictions of the EffNet model, with an optimal sigma fraction, yielding LLL=-7.02 on the train dataset.\n1. A perfect linear decay fit (R^2=1), with an optimal sigma fraction, yielding merely LLL=-6.78 on the train dataset."},{"metadata":{"trusted":true},"cell_type":"code","source":"get_score_graph(fix_rate=0, sigma_fraction=0.11)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"get_score_graph(fix_rate=1, sigma_fraction=0.09)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now the same idea, showing the dependence of the LLL score on the selected sigma fraction."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fix_sigma_pairs = itertools.product([0, 1], np.round(np.arange(0.01, 0.5, 0.01), 2))\nplot_data = pd.Series(index=fix_sigma_pairs, dtype='float', name='LLL')\nplot_data.index = pd.MultiIndex.from_tuples(plot_data.index, names=['fix_rate', 'sigma_fraction'])\nfor fix_rate, sigma_fraction in plot_data.index:\n    _, train_pred = get_score_df(fix_rate, sigma_fraction)\n    lll = laplace_log_likelihood(train_pred['FVC'], train_pred['pred_fvc'], train_pred['pred_confidence'])\n    plot_data.loc[fix_rate, sigma_fraction] = lll\n\nplot_data2 = plot_data.reset_index()\nplot_data2['a_fit'] = plot_data2['fix_rate'].map({0: 'semi-constant', 1: 'perfrect'})\n\nax = sns.lineplot(\n    x='sigma_fraction',\n    y='LLL',\n    hue='a_fit',\n    data=plot_data2\n)\nfor fix_rate, max_loc in plot_data2.groupby('fix_rate')['LLL'].idxmax().iteritems():\n    row = plot_data2.loc[max_loc]\n    arrowprops = dict(facecolor='black', shrink=0.01, width=2, headwidth=9, headlength=9)\n    x, y = row['sigma_fraction'], row['LLL']\n    ax.annotate(f'{row[\"LLL\"]:.4f}', xy=(x, y), xytext=(x+0.15, y-0.1), arrowprops=arrowprops, annotation_clip=True)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Final graph for now - **what would it look like to have a perfect prediction of FVC?**\n\nSo what I did was to use the exact FVC of the final three visits in the train set, and pretend to have predicted them exactly. The lowest possible sigma is 70 (see the Evaluation section of the contest), so an exact prediction with sigma==70 is the best we can possibly do on the training set.\n\nIn addition, I wanted to show two other things:\n1. The effect of sigma on such a perfect prediction (different hues)\n1. The effect of adding noise to the exact prediction of FVC (x axis)\n\nYou can see that the more the FVC deviates from the exact value, the more benefit a higher sigma has. I think that this tradeoff is the whole point of the LLL score."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_pred = train.copy()\nnoise_sigma_pairs = itertools.product(np.round(np.arange(0.0, 0.1, 0.01), 2), range(70, 150, 10))\nplot_data = pd.Series(index=noise_sigma_pairs, dtype='float', name='LLL')\nplot_data.index = pd.MultiIndex.from_tuples(plot_data.index, names=['noise_rate', 'sigma'])\nfor noise_rate, sigma in plot_data.index:\n    noise = np.random.normal(loc=1, scale=noise_rate, size=len(train_pred))\n    train_pred['pred_fvc'] = train_pred['FVC'] * noise\n    train_pred['pred_confidence'] = sigma\n    train_pred = train_pred.groupby('Patient').tail(3)\n    lll = laplace_log_likelihood(train_pred['FVC'], train_pred['pred_fvc'], train_pred['pred_confidence'])\n    plot_data.loc[noise_rate, sigma] = lll\n\nplot_data2 = plot_data.reset_index()\n\nax = sns.lineplot(\n    x='noise_rate',\n    y='LLL',\n    hue='sigma',\n    data=plot_data2,\n    legend='full',\n)\nbest_val = plot_data.loc[0, 70]\narrowprops = dict(facecolor='black', shrink=0.01, width=2, headwidth=9, headlength=9)\nx, y = 0, best_val\nax.annotate(f'Best possible score:\\n{best_val:.4f}', xy=(x, y), xytext=(x+0.05, y-0.3), arrowprops=arrowprops, annotation_clip=True, va='center')\nbest_lb = -6.8305\nax.axhline(best_lb)\nx, y = 0.04, best_lb\nax.annotate(f'Best score on LB:\\n{best_lb:.4f}', xy=(x, y), xytext=(x+0.02, y+1.3), arrowprops=arrowprops, annotation_clip=True, va='center')\nax.legend(loc='upper left', bbox_to_anchor=(1., 1.))\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"To summarize:\n1. The CT images had little-to-no effect on the top-scoring prediction models in this contest.\n1. The use of linear decay as a prediction target was not useful, and *even a perfect prediction* of that target wouldn't have been very beneficial. \n1. A direct prediction of FVC could possibly yield much better results, which is probably why the basic tabular-data models succeeded pretty well, especially when combined with a decent estimation of sigma using the quantiles method. \n1. A perfect prediction in this competition would probably score around LLL=-4.5, which is 1-2 orders of magnitude higher than the range of scores in the top 1000 scoring kernels (between -6.90 and -6.83) - **there is potentially much room to grow here!**\n\nPlease join the discussion: [Some final thoughts on the competition](https://www.kaggle.com/c/osic-pulmonary-fibrosis-progression/discussion/190016)"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}